Bird
Raised Fist0

Query Understanding - Intent Classification, Entity Extraction

SYSTEM-DESIGNmediumInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Query Understanding - Intent Classification, Entity Extraction
Design a system to classify user intent and extract entities from queries.
system-designmediumFreq: highRanking & Retrieval
Prerequisites:TokenizationTransfer Learning and Fine-Tuning
🎯
Problem Framing
Business Objective

Improve user satisfaction by accurately understanding user queries to provide relevant search results.

ML Objective

Classify user intent and extract entities from queries to enhance search relevance.

Constraints
  • Latency SLA: P99 < 100ms for query processing
  • Label availability: Limited labeled data for new intents
  • Cold start: New entities and intents appear frequently
  • Regulatory: Must comply with data privacy regulations
  • Scalability: Handle 10,000 QPS during peak times
Clarifying Questions
  • What are the most common intents we need to classify?
  • How frequently do new entities appear in queries?
  • What is the current latency requirement for query processing?
  • How is labeled data currently obtained and updated?
  • Are there any specific privacy regulations we need to consider?
Out of Scope
✗ Backend search ranking algorithms
✗ User interface design
✗ Non-query based user interactions
📊
Offline Metrics
F1 Score: Balances precision and recall, important for imbalanced intent classes.
Entity Extraction Accuracy: Measures the correctness of extracted entities.
📈
Online Metrics
Query Success Rate: Percentage of queries that return relevant results.
User Engagement: Measured by session length or number of interactions per session.
⚖️
Trade-offs

High F1 score offline might not directly translate to high query success rate online due to differences in user behavior and query context.

💼
Business Metric

User satisfaction score based on feedback and engagement metrics.

🗄️
Data Sources
  • Search logs: User queries and click data, updated in real-time.
  • User feedback: Explicit ratings and implicit signals, updated daily.
  • Knowledge base: Entity definitions and relationships, updated weekly.
🏷️
Data Properties
Labeling Strategy

Implicit feedback from user interactions and explicit human annotations for new intents.

Data Volume

Approximately 100 million queries per day, 10 TB/month.

Freshness

Real-time updates for search logs, T-1 day for user feedback.

⚠️
Data Challenges
  • Sparse labels for new intents
  • Position bias in click data
  • Cold start for new entities
  • Data privacy compliance
🔧
Key Features
embedding
Query Text
Captures semantic meaning of the query.
categorical
User Context
Personalizes intent classification based on user history.
binary
Entity Presence
Indicates presence of known entities in the query.
numerical
Query Length
Short queries might indicate different intents than longer ones.
categorical
Time of Day
Certain intents may be time-dependent.
⚡
Feature Engineering Notes
Real-time vs Batch

Query text and user context need real-time computation; entity presence can be pre-computed.

Feature Store

Required

🐍
Feature Snippet
Python
def compute_query_features(query, user_context):
    query_embedding = embed_query(query)
    entity_presence = check_entity_presence(query)
    return {
        'query_embedding': query_embedding,
        'user_context': user_context,
        'entity_presence': entity_presence
    }
🤖
Model Selection
Baseline

Logistic Regression for intent classification.

Proposed Model

BERT-based model fine-tuned for intent classification and entity extraction.

BERT captures contextual information and can be fine-tuned for specific tasks, handling both intent classification and entity extraction effectively.
✗
Alternatives Rejected
RNN: Rejected due to inability to capture long-range dependencies effectively.
XGBoost: Rejected due to poor performance on text data without extensive feature engineering.
🏋️
Training Strategy

Offline training with periodic updates using new labeled data and user feedback.

🚀
Serving Infrastructure
LatencyP99 < 100ms end-to-end
Throughput10,000 QPS
ArchitectureSingle-stage model serving for simplicity and reduced latency.
CachingCache frequent queries with a TTL of 5 minutes to reduce load.
FallbackRule-based fallback using keyword matching when the model is unavailable.
📡
Data Drift Signals
Change in query length distribution
Shift in entity frequency
New intents appearing in logs
📉
Model Drift Signals
Decrease in F1 Score
Increase in prediction latency
Change in prediction confidence distribution
🔔
Alerting & Retraining
Retraining Trigger

F1 Score drops below 0.8 or significant drift in entity frequency.

Alerting Strategy

On-call engineer paged for latency spikes; automated logs for prediction errors; auto-recovery for minor drifts.

🎯
Interview Angles
  • How to handle cold start for new intents and entities?
  • Balancing latency and accuracy in real-time systems.
  • Strategies for labeling and updating data in dynamic environments.
⚠️
Common Mistakes
Ignoring latency constraints in model selection.
Overfitting to offline metrics without considering online performance.
Neglecting data privacy and compliance requirements.
Failing to implement a robust fallback strategy.
📌
Key Decisions
  • Choosing BERT for its contextual understanding capabilities.
  • Implementing a caching strategy to meet latency requirements.
  • Designing a monitoring system that detects both data and model drift.

Practice

(1/5)
1. For an ad auction system aiming to maximize revenue, which metric should be optimized?
easy
A. Click-through Rate (CTR)
B. Conversion Rate
C. Ad Impressions
D. Expected Revenue

Solution

  1. Step 1: Align metric with objective

    To maximize revenue, the system should optimize for Expected Revenue, which considers both CTR and bid amounts.

  2. Step 2: Eliminate wrong options

    • Click-through Rate (CTR): Measures engagement, not directly linked to revenue.
    • Conversion Rate: Focuses on post-click actions, not directly on revenue.
    • Ad Impressions: Measures visibility, not revenue.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Optimize Expected Revenue for revenue goals. [OK]
Hint: Optimize Expected Revenue for revenue goals. [OK]
Common Mistakes:
2. In a production-scale embedding pipeline for text data, which component is responsible for converting raw text into numerical vectors?
easy
A. Data Loader
B. Feature Extractor
C. Tokenizer
D. Batch Normalizer

Solution

  1. Step 1: Identify the role of each component.

    The Tokenizer is responsible for converting raw text into tokens, which are then converted into numerical vectors.

  2. Step 2: Eliminate wrong options.

    Data Loader handles data input, Feature Extractor processes features but not directly from raw text, and Batch Normalizer normalizes data batches.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Tokenizer converts text to tokens. [OK]
Hint: Tokenizer converts text to tokens. [OK]
Common Mistakes:
3. A search ranking system suddenly shows irrelevant results at the top. What is the most likely failure mode?
medium
A. Query-document feature mismatch
B. Data leakage from training to test set
C. Overfitting due to too many features
D. Index corruption

Solution

  1. Step 1: Analyze the symptom

    Irrelevant results suggest a mismatch in understanding query-document relevance.

  2. Step 2: Identify the most likely cause

    Feature mismatch can lead to poor relevance scoring.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Feature mismatch leads to poor relevance. [OK]
Hint: Feature mismatch leads to poor relevance. [OK]
Common Mistakes:
4. In a two-stage retrieval system, when should you prioritize a simpler candidate generation model over a complex one?
medium
A. When the ranking stage is computationally expensive and needs more resources.
B. When precision is more important than recall in the candidate generation stage.
C. When the dataset is small and computation resources are abundant.
D. When the ranking model is not capable of handling large candidate sets.

Solution

  1. Step 1: Consider resource allocation

    If the ranking stage is computationally expensive, a simpler candidate generation model can save resources for the ranking stage.

  2. Step 2: Eliminate wrong options

    When the dataset is small and computation resources are abundant. is incorrect because a small dataset doesn't necessitate a simpler model. When precision is more important than recall in the candidate generation stage. is incorrect because recall is typically prioritized in candidate generation. When the ranking model is not capable of handling large candidate sets. is incorrect because the ranking model should handle the candidate set size.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Save resources for ranking if it's costly. [OK]
Hint: Save resources for ranking if it's costly. [OK]
Common Mistakes:
5. When deploying a Two-Tower Model for Retrieval, which approach is better for handling cold-start users and why?
medium
A. Use random embeddings until enough data is collected.
B. Use collaborative filtering to generate initial embeddings.
C. Use content-based features to generate initial embeddings.
D. Use embeddings from a pre-trained language model.

Solution

  1. Step 1: Consider the cold-start problem

    Cold-start users lack interaction data, making content-based features useful for initial embeddings.

  2. Step 2: Eliminate wrong options

    Use collaborative filtering to generate initial embeddings. relies on interaction data, which cold-start users lack. Use random embeddings until enough data is collected. provides no meaningful information. Use embeddings from a pre-trained language model. might not capture user-specific features effectively.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Content-based features help with cold-start. [OK]
Hint: Content-based features help with cold-start. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard