Bird
Raised Fist0

Implement Confusion Matrix from Predictions

CODINGbeginnerInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Confusion Matrix from Predictions vs Confusion Matrix from True Labels

When evaluating a model's performance, would you compute the confusion matrix from predictions or directly from true labels? Justify your choice.

📊
Comparison
DimensionConfusion Matrix from PredictionsConfusion Matrix from True Labels
Accuracy EvaluationHighLow
Baseline AssessmentLowHigh
Real-time UsabilityMediumLow
Sensitivity to ErrorsHighLow
Class Imbalance HandlingLowHigh
✅
Verdict
Default to using confusion matrices from predictions when evaluating model performance in controlled environments with available and accurate predictions. Switch to true labels when predictions are unavailable or when assessing data distribution and class imbalance before training.
💡
Key Insight
Choose confusion matrices from predictions for performance evaluation and from true labels for baseline and distribution insights.
⚠️
Interview Trap
❌ Wrong: Always use predictions for confusion matrices because they show model performance.
This ignores scenarios where predictions are unavailable or unreliable, and the need for baseline assessments.
✅ Consider the availability and reliability of predictions, and whether you need a baseline assessment or real-time evaluation.
Confusion Matrix from Predictions
"Directly evaluates model output accuracy."
Weaknesses
  • Requires model predictions, which may not always be available.
  • Sensitive to prediction errors, leading to misleading results if predictions are poor.
  • Does not provide insights into model bias or variance.
  • Less useful for models with high class imbalance.
Ideal When
  • When predictions are readily available.
  • When n < 10K and predictions are accurate.
  • When evaluating model performance in a controlled environment.
  • When class distribution is balanced.
Google uses confusion matrices from predictions to evaluate the accuracy of their image classification models during testing phases.
Confusion Matrix from True Labels
"Provides a baseline for model evaluation without needing predictions."
Weaknesses
  • Cannot evaluate model performance directly.
  • Requires additional steps to compare with predictions.
  • Not useful for real-time evaluation.
  • Does not account for prediction errors.
Ideal When
  • When predictions are not available.
  • When n > 10K and you need a baseline for comparison.
  • When evaluating data distribution before model training.
  • When class imbalance needs to be assessed.
Amazon uses confusion matrices from true labels to understand data distribution before deploying models in production.
🐍
Confusion Matrix from Predictions - Example
▼ Show
Python
import numpy as np

def confusion_matrix_from_predictions(y_true, y_pred, num_classes):
    cm = np.zeros((num_classes, num_classes), dtype=int)
    for t, p in zip(y_true, y_pred):
        cm[t, p] += 1
    return cm

# Example usage
y_true = [0, 1, 2, 2, 0]
y_pred = [0, 2, 1, 2, 0]
print(confusion_matrix_from_predictions(y_true, y_pred, 3))
🐍
Confusion Matrix from True Labels - Example
▼ Show
Python
import numpy as np

def confusion_matrix_from_labels(y_true, num_classes):
    cm = np.zeros((num_classes, num_classes), dtype=int)
    for t in y_true:
        cm[t, t] += 1
    return cm

# Example usage
y_true = [0, 1, 2, 2, 0]
print(confusion_matrix_from_labels(y_true, 3))
✅
Use Confusion Matrix from Predictions When
  • •When predictions are available and accurate.
  • •When n < 10K and class distribution is balanced.
  • •When evaluating model performance in a controlled test environment.
✅
Use Confusion Matrix from True Labels When
  • •When predictions are not available.
  • •When n > 10K and you need a baseline for comparison.
  • •When assessing class imbalance before model training.
↳
Nuances
  • For small datasets with balanced classes, predictions provide a clearer performance picture.
  • True labels are essential for understanding data distribution before model training.
  • In high class imbalance scenarios, true labels offer better insights into potential model biases.
  • Predictions can be misleading if the model is not well-calibrated, especially in early development stages.

Practice

(1/5)
1. 
import numpy as np

def cosine_similarity(a, b):
    dot_product = np.dot(a, b)
    norm_a = np.linalg.norm(a)
    norm_b = np.linalg.norm(b)
    return dot_product / (norm_a * norm_b)

vector1 = np.array([1, 2, 3])
vector2 = np.array([4, 5, 6])

result = cosine_similarity(vector1, vector2)
print(result)
easy
A. 0.9746318461970762
B. 0.9999999999999998
C. 1.0
D. 0.0

Solution

  1. Step 1: Calculate dot product and norms

    Dot product = 32, norm_a = 3.7416573867739413, norm_b = 8.774964387392123.

  2. Step 2: Compute cosine similarity

    Cosine similarity = 32 / (3.7416573867739413 * 8.774964387392123) = 0.9746318461970762.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Dot product over product of norms. [OK]
Hint: Dot product over product of norms. [OK]
Common Mistakes:
2. In a pipeline that includes flipping, cropping, and normalizing images, where should normalization be applied for best results?
medium
A. Before flipping and cropping
B. After flipping but before cropping
C. After cropping but before flipping
D. After both flipping and cropping

Solution

  1. Step 1: Understand the impact of operations

    Normalization should be the last step to ensure consistent scaling.

  2. Step 2: Evaluate each option

    Applying normalization last ensures it accounts for all transformations.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Normalize last for consistent scaling. [OK]
Hint: Normalize last for consistent scaling. [OK]
Common Mistakes:
3. 
import numpy as np

# Backpropagation for a single neuron
inputs = np.array([1.0, 2.0, 3.0])
weights = np.array([0.2, 0.8, -0.5])
bias = 2.0
learning_rate = 0.01

d_output = np.array([1.0])  # Gradient of loss w.r.t output

# Incorrect gradient calculation
weights_gradient = d_output * inputs
weights -= learning_rate * weights_gradient
print(weights)
medium
A. Weights are updated correctly.
B. Weights are updated incorrectly due to incorrect learning rate.
C. Weights are not updated; learning rate is too small.
D. Weights are updated incorrectly due to missing chain rule.

Solution

  1. Step 1: Identify error in gradient calculation

    The code incorrectly applies the gradient directly without considering the chain rule.

  2. Step 2: Correct approach

    Should multiply by the derivative of the activation function.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Always apply chain rule in backpropagation [OK]
Hint: Always apply chain rule in backpropagation [OK]
Common Mistakes:
4. In the following non-max suppression code, the function sometimes returns an empty list even when there are valid boxes. What is the likely cause? 
def non_max_suppression(boxes, scores, iou_threshold):
    indices = np.argsort(scores)[::-1]
    keep = []
    while len(indices) > 0:
        current = indices[0]
        keep.append(current)
        indices = indices[1:]
        indices = [i for i in indices if iou(boxes[current], boxes[i]) < iou_threshold]
    return keep
medium
A. The boxes array is empty.
B. The scores array is not sorted correctly.
C. The IoU function is incorrect.
D. The IoU threshold is set too high.

Solution

  1. Step 1: Analyze the condition for keeping boxes

    If the IoU threshold is too high, no boxes will be kept.

  2. Step 2: Eliminate other causes

    Sorting and empty boxes would cause different errors; incorrect IoU would not consistently return empty.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    High IoU threshold filters all boxes [OK]
Hint: High IoU threshold filters all boxes [OK]
Common Mistakes:
5. 
import numpy as np

def ndcg_at_k(r, k):
    dcg_max = dcg_at_k(sorted(r, reverse=True), k)
    if not dcg_max:
        return 0.
    return dcg_at_k(r, k) / dcg_max

relevance_scores = [3, 2, 3, 0, 1, 2]
k = 3
print(ndcg_at_k(relevance_scores, k))
The output is unexpectedly low. What is the most likely cause?
medium
A. The k value is too low.
B. The DCG is calculated incorrectly.
C. The relevance scores are not normalized.
D. The ideal DCG is not calculated correctly.

Solution

  1. Step 1: Check Ideal DCG Calculation

    The ideal DCG should be calculated using the sorted relevance scores.

  2. Step 2: Identify the Error

    If the ideal DCG is not calculated correctly, the NDCG will be incorrect.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Check ideal DCG calculation. [OK]
Hint: Check ideal DCG calculation. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard