Bird
Raised Fist0

The Misleading Metrics Mystery

CODINGbeginnerInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Misleading Metrics Mystery
The issue started after the last model update two weeks ago.

A binary classification model used for fraud detection in a financial application, processing 500K transactions daily.

Symptom
Precision and recall metrics reported by the system are unexpectedly low despite high accuracy.
📡
Signals
metric
Precision metric reported by the system
Precision dropped from 0.85 to 0.45 after the update
metric
Recall metric reported by the system
Recall dropped from 0.80 to 0.40 after the update
metric
Overall accuracy of the model
Accuracy remains high at 0.95
code
Code snippet for precision calculation
precision = tp / (tp + fp)
code
Code snippet for recall calculation
recall = tp / (tp + fn)
🔬
Hypotheses
The confusion matrix is being calculated incorrectly.
Supporting Signals
Precision droppedRecall dropped
Verify: Check the confusion matrix calculation logic.
print(confusion_matrix(y_true, y_pred))
The precision and recall calculations are using incorrect values.
Supporting Signals
Code snippet for precision calculationCode snippet for recall calculation
Verify: Verify the values used in precision and recall calculations.
print(f'TP: {tp}, FP: {fp}, FN: {fn}')
🎯
Root Cause
The precision and recall calculations were using incorrect values for true positives, false positives, and false negatives.
Why It Happened

A recent code refactor mistakenly swapped the indices used to extract values from the confusion matrix.

🔧
Fix Strategy
ImmediateRevert to the previous version of the model metrics calculation.
Short-termCorrect the indices used in the confusion matrix to calculate precision and recall.
Long-termImplement unit tests for metric calculations to ensure correctness after code changes.
🐍
Python Fix
Python
tp, fp, fn = confusion_matrix[1, 1], confusion_matrix[0, 1], confusion_matrix[1, 0]
precision = tp / (tp + fp)
recall = tp / (tp + fn)
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting to the training data.
Why candidates think this: The high accuracy might suggest overfitting.
Why wrong: Overfitting would not cause precision and recall to drop while accuracy remains high.
The data distribution has changed significantly.
Why candidates think this: A change in data distribution could affect model performance.
Why wrong: Data distribution changes would likely affect accuracy as well, not just precision and recall.
📊
Monitoring to Add
  • Monitor precision and recall metrics for significant drops.
  • Set up alerts for discrepancies between accuracy and precision/recall.
  • Track changes in confusion matrix values over time.
📋
Process Change

Require code reviews for all metric calculation changes.

🏗️
Architectural Change

Use a metrics library that abstracts confusion matrix calculations to reduce manual errors.

💡
Key Insight
Precision and recall drops with stable accuracy often indicate calculation errors, not model performance issues.

Practice

(1/5)
1. What is the time complexity of cropping an image of size n x n to a smaller size m x m using NumPy slicing?
easy
A. O(n^2)
B. O(1)
C. O(n*m)
D. O(m^2)

Solution

  1. Step 1: Understand NumPy slicing

    NumPy slicing creates a view, not a copy, so it operates in constant time.

  2. Step 2: Evaluate the complexity

    The operation does not depend on the size of the arrays but on the slicing indices.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    NumPy slicing is O(1) due to views. [OK]
Hint: NumPy slicing is O(1) due to views. [OK]
Common Mistakes:
2. You implemented a 2D convolution but the output is all zeros. What is the most likely cause?
medium
A. The kernel is not normalized.
B. The kernel is filled with zeros.
C. The image is padded incorrectly.
D. The stride is set too high.

Solution

  1. Step 1: Identify the zero output cause

    If the kernel is filled with zeros, every convolution operation will result in zero, regardless of the image content.

  2. Step 2: Eliminate wrong options

    The kernel is not normalized. affects scaling, not zero output. The image is padded incorrectly. affects boundary handling, not zero output. The stride is set too high. affects output size, not zero output.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Zero kernel leads to zero output. [OK]
Hint: Zero kernel leads to zero output. [OK]
Common Mistakes:
3. In a recommendation system, why might you choose NDCG over precision at k?
medium
A. NDCG is easier to compute.
B. NDCG considers the position of relevant items.
C. NDCG is less sensitive to irrelevant items.
D. NDCG is more interpretable than precision.

Solution

  1. Step 1: Understand NDCG vs Precision

    NDCG accounts for the position of relevant items, giving higher scores to relevant items at higher ranks.

  2. Step 2: Evaluate Options

    Precision at k does not consider the order of items, only their presence.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    NDCG values position. [OK]
Hint: NDCG values position. [OK]
Common Mistakes:
4. You implemented PCA but the variance explained by the first principal component is unexpectedly low. What is the most likely cause?
medium
A. Data was not centered before computing the covariance matrix.
B. Eigenvectors were not normalized.
C. Covariance matrix was computed using rowvar=True.
D. Too few principal components were selected.

Solution

  1. Step 1: Identify the impact of centering data

    Centering data is crucial for PCA as it ensures the covariance matrix reflects true variance.

  2. Step 2: Eliminate wrong options

    Eigenvectors were not normalized. is incorrect because normalization affects direction, not variance. Covariance matrix was computed using rowvar=True. affects orientation, not variance. Too few principal components were selected. is irrelevant to the variance of the first component.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Center data before PCA to capture variance. [OK]
Hint: Center data before PCA to capture variance. [OK]
Common Mistakes:
5. How can you optimize the train-test split with stratification for a very large dataset to reduce memory usage?
hard
A. Use in-place operations to modify the dataset
B. Use a smaller data type for the labels
C. Use a generator to yield batches of data
D. Increase the test size to reduce train set size

Solution

  1. Step 1: Identify memory optimization techniques

    Using a generator allows processing data in batches, reducing memory footprint.

  2. Step 2: Evaluate options

    In-place operations and smaller data types help, but generators are more effective for large datasets.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Generators reduce memory by processing in batches. [OK]
Hint: Generators reduce memory by processing in batches. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard