Bird
Raised Fist0

Ad Auction System - Ranking Ads by Expected Value

SYSTEM-DESIGNhardInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Thursday Latency Spike
Started last Thursday, ongoing for 5 days.

An ad auction system ranking ads by expected value, serving 5M ad impressions per day for a large e-commerce platform.

Symptom
AUC dropped from 0.85 to 0.72 over the past week, and recommendation diversity collapsed by 30%.
📡
Signals
metric
Live AUC observed in production dashboards
Dropped from 0.85 to 0.72 since last Thursday
distribution
Distribution of ad click-through rates (CTR)
CTR variance reduced by 40% since last Thursday
latency
Real-time feature extraction latency
Spiked from P99 < 50ms to P99 = 200ms since last Thursday
feature_importance
Feature importance for 'user_last_active_days'
Dropped from 0.18 to 0.02 after last Thursday's deploy
log
Error logs from feature extraction service
Increased frequency of 'timeout' errors since last Thursday
🔬
Hypotheses
Feature extraction service is causing latency spikes, affecting model performance.
Supporting Signals
Real-time feature extraction latency spikedError logs show increased 'timeout' errors
Verify: Check service logs for timeout patterns and correlate with latency spikes.
import pandas as pd
logs = pd.read_csv('service_logs.csv')
timeouts = logs[logs['error'] == 'timeout']
timeouts.groupby('timestamp').size().plot()
Recent deploy introduced a bug affecting feature importance calculation.
Supporting Signals
Feature importance for 'user_last_active_days' dropped significantly
Verify: Review the code changes in the last deploy for feature importance calculation.
import difflib
with open('old_version.py') as f1, open('new_version.py') as f2:
    diff = difflib.unified_diff(f1.readlines(), f2.readlines())
    for line in diff:
        print(line)
Data distribution shift due to external factors affecting CTR variance.
Supporting Signals
CTR variance reduced significantly
Verify: Analyze CTR distribution over time to identify shifts.
import matplotlib.pyplot as plt
import pandas as pd
ctr_data = pd.read_csv('ctr_data.csv')
ctr_data['date'] = pd.to_datetime(ctr_data['date'])
ctr_data.groupby('date')['ctr'].var().plot()
plt.show()
🎯
Root Cause
Feature extraction service latency spike due to inefficient code changes in the last deploy.
Why It Happened

A recent deployment introduced inefficient code in the feature extraction service, causing increased latency and timeout errors.

🔧
Fix Strategy
ImmediateRollback to the previous stable version of the feature extraction service.
Short-termOptimize the code in the feature extraction service to reduce latency.
Long-termImplement a robust testing framework for feature extraction service to catch latency issues before deployment.
🐍
Python Fix
Python
# Example of optimizing feature extraction code
import time

def optimized_feature_extraction(data):
    # Optimized logic here
    time.sleep(0.01)  # Simulate reduced latency
    return data

# Apply optimized function
raw_data = []  # Assuming raw_data is defined elsewhere
optimized_data = optimized_feature_extraction(raw_data)
✗
Common Wrong Diagnoses
▼ Show
Model overfitting due to insufficient training data.
Why candidates think this: The AUC drop suggests a model performance issue.
Why wrong: The stable validation AUC indicates the model itself is not overfitting.
Data distribution shift due to external market changes.
Why candidates think this: The reduced CTR variance suggests a shift in user behavior.
Why wrong: The timing of the issue aligns with the recent deploy, not external factors.
Feature importance recalculation bug.
Why candidates think this: The drop in feature importance suggests a calculation error.
Why wrong: The issue is related to latency, not the calculation logic itself.
📊
Monitoring to Add
  • Real-time latency monitoring for feature extraction service
  • Alert on sudden drops in feature importance
  • Monitor CTR variance for unexpected shifts
  • Error rate monitoring for feature extraction service
📋
Process Change

Introduce a mandatory code review process focusing on performance implications before deployment.

🏗️
Architectural Change

Decouple feature extraction service from the main pipeline to allow independent scaling and optimization.

💡
Key Insight
Latency spikes in feature extraction can cascade into severe model performance degradation, highlighting the need for robust latency monitoring.

Practice

(1/5)
1. In a search ranking system, which component is primarily responsible for determining the relevance of a document to a query?
easy
A. Query parser
B. Ranking algorithm
C. Indexing module
D. Crawler

Solution

  1. Step 1: Identify the role of each component

    The ranking algorithm is responsible for determining the relevance of documents to a query.

  2. Step 2: Eliminate wrong options

    Indexing module organizes data, query parser interprets queries, and crawler collects data.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Ranking algorithm determines relevance. [OK]
Hint: Ranking algorithm determines relevance. [OK]
Common Mistakes:
2. A bot detection system using device fingerprinting is failing to detect bots effectively. What is the most likely failure mode?
medium
A. The system is not considering user agent strings.
B. The system is not updating fingerprints frequently enough.
C. The system is overly reliant on IP addresses.
D. The system is using outdated behavioral models.

Solution

  1. Step 1: Identify the core issue with device fingerprinting.

    Device fingerprints need frequent updates to remain effective against evolving bots.

  2. Step 2: Eliminate wrong options.

    Behavioral models and IP reliance are not directly related to fingerprinting; user agent strings are not unique enough.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Frequent updates needed for effective fingerprinting [OK]
Hint: Frequent updates needed for effective fingerprinting [OK]
Common Mistakes:
3. In an ML monitoring platform, what is a likely failure mode if the alerting system is not properly calibrated?
medium
A. The retraining process will be triggered too frequently
B. The system will miss significant drifts
C. The model's accuracy will degrade over time
D. The model will overfit to the training data

Solution

  1. Step 1: Understand the role of alerting calibration

    Proper calibration ensures alerts are meaningful and not overly sensitive.

  2. Step 2: Identify the failure mode

    If not calibrated, the alerting system may trigger retraining too often, leading to unnecessary computational costs.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Over-alerting = Frequent retraining [OK]
Hint: Over-alerting = Frequent retraining [OK]
Common Mistakes:
4. In a semantic search system using a Bi-Encoder, what is a likely failure mode if the document embeddings are not updated regularly?
medium
A. Increased computational cost during retrieval
B. Increased latency in query processing
C. Decreased retrieval accuracy due to outdated embeddings
D. Overfitting to the training data

Solution

  1. Step 1: Identify the impact of outdated embeddings

    Outdated embeddings can lead to decreased retrieval accuracy as they may not reflect current document content.

  2. Step 2: Eliminate wrong options

    Increased computational cost and latency are not directly related to outdated embeddings, and overfitting is a training issue.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Outdated embeddings = accuracy drop [check]
Hint: Outdated embeddings = accuracy drop [check]
Common Mistakes:
5. As the number of users and items grows significantly, which component of a Two-Tower Model for Retrieval is likely to become a bottleneck first?
hard
A. Similarity score computation
B. Item embedding computation
C. User embedding computation
D. Model training time

Solution

  1. Step 1: Consider the scaling impact on components

    As users and items grow, computing similarity scores between all pairs becomes computationally expensive.

  2. Step 2: Eliminate wrong options

    User embedding computation and B scale linearly with users and items. Model training time is affected but not as immediately as similarity computation.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Similarity computation scales poorly with size. [OK]
Hint: Similarity computation scales poorly with size. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard