Bird
Raised Fist0

Classification Metrics - Precision, Recall, F1, When Each Matters

THEORYbeginnerInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Recall Vanishing Act
Issue started after the latest model update last Monday.

Spam detection model for an email service provider, processing 5 million emails daily.

Symptom
Model accuracy is stable at 95%, but recall has dropped to 0.
📡
Signals
metric
Recall metric from the model evaluation dashboard
Recall dropped from 0.85 to 0.00 after the latest update
metric
Precision metric from the model evaluation dashboard
Precision increased from 0.90 to 0.98 after the latest update
feature_importance
Feature importance ranking from the model
Feature 'email_length' importance dropped from 0.15 to 0.01
distribution
Distribution of predicted probabilities for the positive class
All predicted probabilities are below 0.1
log
Model training log
No errors or warnings during training
🔬
Hypotheses
The model's threshold for classifying spam is too high.
Supporting Signals
Recall dropped to 0All predicted probabilities are below 0.1
Verify: Check the threshold used for classification in the model deployment script.
threshold = 0.9; predictions = (model.predict_proba(X)[:, 1] > threshold).astype(int)
A critical feature was inadvertently removed or its influence reduced.
Supporting Signals
Feature 'email_length' importance dropped significantly
Verify: Review the feature engineering pipeline for recent changes.
df[['email_length']].describe()
🎯
Root Cause
The classification threshold was inadvertently set too high during deployment.
Why It Happened

A configuration error in the deployment script set the threshold to 0.9 instead of the intended 0.5.

🔧
Fix Strategy
ImmediateRollback to the previous model version with the correct threshold.
Short-termCorrect the threshold setting in the deployment script and redeploy.
Long-termImplement a configuration management system to prevent manual errors in deployment scripts.
🐍
Python Fix
Python
threshold = 0.5; predictions = (model.predict_proba(X)[:, 1] > threshold).astype(int)
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting to the training data.
Why candidates think this: The high precision might suggest overfitting to specific patterns.
Why wrong: Overfitting would typically affect both precision and recall, not just recall.
A bug in the feature engineering pipeline.
Why candidates think this: The drop in feature importance suggests a feature issue.
Why wrong: The feature importance drop was a result of the threshold issue, not the cause.
📊
Monitoring to Add
  • Alert if recall drops below 0.5 in production
  • Monitor distribution of predicted probabilities for anomalies
  • Check for significant changes in feature importance post-deployment
📋
Process Change

Introduce a deployment checklist that includes verification of critical parameters like thresholds.

🏗️
Architectural Change

Automate deployment configurations using a CI/CD pipeline with parameter validation.

💡
Key Insight
A misconfigured threshold can drastically affect recall without impacting precision, highlighting the importance of correct parameter settings in deployment.

Practice

(1/5)
1. A transformer model is underperforming on a translation task. The model seems to ignore the context of the input sequence. What is the most likely cause?
medium
A. The model's attention heads are not properly initialized.
B. The model's positional encodings are not being used.
C. The model's feedforward layers are too shallow.
D. The model's dropout rate is too high.

Solution

  1. Step 1: Identify the role of positional encodings.

    Positional encodings provide sequence order information, crucial for context.

  2. Step 2: Eliminate wrong options.

    The model's attention heads are not properly initialized. is incorrect as initialization affects convergence, not context. The model's feedforward layers are too shallow. is incorrect because feedforward depth affects capacity, not context. The model's dropout rate is too high. is incorrect as dropout affects regularization, not context.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Positional encodings ensure context. [OK]
Hint: Positional encodings ensure context. [OK]
Common Mistakes:
2. A neural network is not learning effectively, and the gradients are extremely small. What is the most likely cause?
medium
A. The learning rate is too high.
B. The network is too shallow.
C. The activation function is causing vanishing gradients.
D. The dataset is too small.

Solution

  1. Step 1: Identify symptoms of vanishing gradients

    Extremely small gradients often indicate the vanishing gradient problem, typically caused by certain activation functions like sigmoid or tanh.

  2. Step 2: Eliminate wrong options

    The learning rate is too high. is incorrect because a high learning rate would cause divergence. The network is too shallow. is incorrect because a shallow network is less prone to vanishing gradients. The dataset is too small. is unrelated to gradient size.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Small gradients often mean vanishing issue. [OK]
Hint: Small gradients often mean vanishing issue. [OK]
Common Mistakes:
3. When evaluating a recommendation system, why might you choose NDCG over MAP?
medium
A. NDCG is easier to compute than MAP.
B. NDCG is more interpretable than MAP.
C. NDCG accounts for the position of relevant items, while MAP does not.
D. NDCG is less sensitive to the number of relevant items than MAP.

Solution

  1. Step 1: Understand metric differences

    NDCG considers the position of relevant items, giving higher weight to top-ranked items.

  2. Step 2: Eliminate wrong options

    MAP considers all relevant items equally, NDCG is not necessarily easier or more interpretable, and both metrics are sensitive to the number of relevant items.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    NDCG weights top positions more. [OK]
Hint: NDCG weights top positions more. [OK]
Common Mistakes:
4. You need to apply model interpretability techniques to a real-time fraud detection system. Which approach is most suitable?
hard
A. Use attention mechanisms in the model architecture
B. Use LIME for batch processing of transactions
C. Use SHAP for every transaction in real-time
D. Use a simple decision tree for interpretability

Solution

  1. Step 1: Consider real-time constraints.

    Real-time systems require fast interpretability methods.

  2. Step 2: Evaluate each option.

    SHAP and LIME are computationally expensive for real-time use. Attention mechanisms provide built-in interpretability without significant overhead.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Attention is efficient for real-time interpretability. [OK]
Hint: Attention is efficient for real-time interpretability. [OK]
Common Mistakes:
5. You need to design a system using RAG prompting for a real-time question-answering application. Which approach ensures the system remains efficient and accurate?
hard
A. Use a static set of documents for retrieval to ensure consistency.
B. Use a large, diverse document set without updates to avoid bias.
C. Continuously update the document set based on user interactions.
D. Rely solely on the model's pre-trained knowledge without retrieval.

Solution

  1. Step 1: Understand RAG efficiency

    For real-time applications, updating documents based on interactions ensures relevance and accuracy.

  2. Step 2: Eliminate wrong options

    Use a static set of documents for retrieval to ensure consistency. lacks adaptability. Rely solely on the model's pre-trained knowledge without retrieval. ignores the retrieval component. Use a large, diverse document set without updates to avoid bias. risks outdated information.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Update documents for real-time RAG efficiency. [OK]
Hint: Update documents for real-time RAG efficiency. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard