Bird
Raised Fist0

Visual Search System - Image Embedding, Approximate NN, Serving

SYSTEM-DESIGNhardInterview freq: medium
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Visual Search System - Image Embedding, Approximate NN, Serving
Design a scalable visual search system using image embeddings and approximate nearest neighbors.
system-designhardFreq: mediumPrediction Systems
Prerequisites:Convolutional Neural NetworksEmbeddingsVector Search and ANN
🎯
Problem Framing
Business Objective

Increase user engagement by providing fast and accurate image search results.

ML Objective

Maximize the relevance of retrieved images given a query image using image embeddings.

Constraints
  • P99 latency < 100ms for search results
  • High throughput to handle 10,000 QPS
  • Cold start problem for new images
  • Limited labeled data for training
  • Scalability to millions of images
Clarifying Questions
  • What is the expected scale of the image database?
  • Are there any specific latency requirements?
  • How frequently is the image database updated?
  • What is the primary metric for success?
  • Are there any constraints on computational resources?
Out of Scope
✗ Text-based search
✗ Video search
✗ Real-time image processing
📊
Offline Metrics
Mean Average Precision (mAP) because it evaluates the precision of retrieved images across multiple queries.
Recall@K to ensure a high percentage of relevant images are retrieved within the top K results.
📈
Online Metrics
Click-through Rate (CTR) to measure user engagement with search results.
Time to First Byte (TTFB) to ensure low latency in serving results.
⚖️
Trade-offs

High mAP might not always correlate with high CTR if the retrieved images are relevant but not engaging. Balancing precision and user engagement is crucial.

💼
Business Metric

User engagement as measured by session duration and repeat visits.

🗄️
Data Sources
  • User-uploaded images (real-time)
  • Product catalog images (updated daily)
  • Third-party image databases (updated weekly)
🏷️
Data Properties
Labeling Strategy

Implicit feedback from user interactions (clicks, time spent) to infer relevance.

Data Volume

Approximately 10 million images with 1 TB of new data per month.

Freshness

Features should be updated daily to incorporate new images and user interactions.

⚠️
Data Challenges
  • Sparse labels due to implicit feedback
  • Cold start for new images with no interactions
  • Distribution shift as new images are added
  • Scalability to handle large image datasets
🔧
Key Features
feat
feat
feat
feat
feat
⚡
Feature Engineering Notes
Real-time vs Batch

Image embeddings are pre-computed in batch, while user interaction features are updated in real-time.

Feature Store

Required

🐍
Feature Snippet
Python
def compute_image_embedding(image):
    model = load_pretrained_model()
    embedding = model.extract_features(image)
    return embedding
🤖
Model Selection
Baseline

Simple nearest neighbor search using raw pixel values.

Proposed Model

Convolutional Neural Network (CNN) for image embeddings combined with Approximate Nearest Neighbor (ANN) search.

CNNs effectively capture visual features, and ANN provides scalable and fast retrieval.
✗
Alternatives Rejected
Logistic Regression: Rejected due to inability to handle high-dimensional image data.
XGBoost: Rejected because it requires feature engineering that doesn't capture complex image patterns.
Exact Nearest Neighbor: Rejected due to high latency and computational cost at scale.
🏋️
Training Strategy

Offline training with periodic updates to incorporate new data and improve embeddings.

🚀
Serving Infrastructure
LatencyP99 < 100ms end-to-end
Throughput10,000 QPS
ArchitectureTwo-stage (retrieval + ranking) to first narrow down candidates and then rank them for relevance.
CachingCache popular queries with a TTL of 1 hour to reduce computation.
FallbackUse cached results or a rule-based system if the model is unavailable.
📡
Data Drift Signals
Shift in image embedding distributions
Change in user interaction patterns
Increase in cache miss rate
📉
Model Drift Signals
Decrease in mAP over time
Increase in latency for ANN search
Drop in CTR
🔔
Alerting & Retraining
Retraining Trigger

Retrain when mAP drops by more than 5% over a month.

Alerting Strategy

Page on-call engineer if latency exceeds 150ms or if CTR drops by 10% week-over-week.

🎯
Interview Angles
  • How to handle cold start for new images?
  • Balancing precision and recall in image retrieval.
  • Scalability challenges with large image datasets.
⚠️
Common Mistakes
Ignoring latency constraints in model selection.
Overfitting to offline metrics without considering online performance.
Neglecting the cold start problem for new images.
Failing to address data freshness and update frequency.
📌
Key Decisions
  • Choosing CNNs for embedding due to their ability to capture complex image features.
  • Implementing a two-stage retrieval and ranking architecture for efficiency.
  • Using implicit feedback for labeling to leverage user interactions.

Practice

(1/5)
1. In an ad auction system, which component is responsible for estimating the expected value of an ad based on historical click-through rates and bid amounts?
easy
A. User Targeting Engine
B. Bid Management System
C. Ad Ranking Module
D. Ad Inventory Manager

Solution

  1. Step 1: Identify the component's role

    The Ad Ranking Module is responsible for estimating the expected value of an ad by considering factors like historical click-through rates and bid amounts.

  2. Step 2: Eliminate wrong options

    • Bid Management System: Focuses on managing bids, not estimating ad value.
    • User Targeting Engine: Targets users, not involved in value estimation.
    • Ad Inventory Manager: Manages ad inventory, not value estimation.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Ad Ranking estimates value. [OK]
Hint: Ad Ranking Module estimates ad value. [OK]
Common Mistakes:
2. In the context of avoiding filter bubbles in recommendation systems, which component is responsible for ensuring content diversity?
easy
A. Collaborative filtering
B. Diversity re-ranking layer
C. User profiling
D. Content-based filtering

Solution

  1. Step 1: Identify the role of each component

    Collaborative and content-based filtering focus on user preferences, while diversity re-ranking ensures varied content.

  2. Step 2: Eliminate wrong options

    Collaborative filtering (A) and content-based filtering (B) do not inherently ensure diversity. User profiling (D) focuses on understanding user preferences, not diversity.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Diversity re-ranking ensures varied content [OK]
Hint: Diversity re-ranking ensures varied content [OK]
Common Mistakes:
3. For a spam detection system, which metric is most appropriate to optimize for?
easy
A. F1 Score
B. Precision
C. Recall
D. Accuracy

Solution

  1. Step 1: Understand the importance of balancing precision and recall

    F1 Score balances precision and recall, which is crucial in spam detection.

  2. Step 2: Eliminate less balanced metrics

    Accuracy (A) can be misleading with imbalanced data. Precision (B) and Recall (C) alone don't balance false positives and negatives.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    F1 balances precision and recall. [OK]
Hint: F1 balances precision and recall. [OK]
Common Mistakes:
4. As the user base grows 10x, which component of a hate speech detection system is most likely to become a bottleneck?
hard
A. Model Training
B. Data Preprocessing
C. Feature Extraction
D. Model Inference

Solution

  1. Step 1: Consider the impact of scale

    With a 10x increase in users, the system must handle significantly more inference requests.

  2. Step 2: Eliminate wrong options

    Data preprocessing and feature extraction are typically batch processes, and model training is not affected by user scale.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    10x users = Inference bottleneck [OK]
Hint: 10x users = Inference bottleneck [OK]
Common Mistakes:
5. As the dataset size increases 10x in a FAISS-based vector search system, which component is most likely to become a bottleneck first?
hard
A. Index storage
B. Search algorithm
C. Dimensionality reduction
D. Quantization process

Solution

  1. Step 1: Consider the impact of dataset size increase.

    Index storage will grow significantly, potentially becoming a bottleneck.

  2. Step 2: Eliminate wrong options.

    Search algorithm, dimensionality reduction, and quantization are less affected by size increase compared to storage.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Index storage grows with dataset size. [OK]
Hint: Index storage grows with dataset size. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard