Bird
Raised Fist0

Handling Class Imbalance - Oversampling, Undersampling, Focal Loss

THEORYintermediateInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Oversampling vs Focal Loss

In a dataset with a severe class imbalance, would you choose oversampling or focal loss to improve model performance? Justify your choice.

📊
Comparison
DimensionOversamplingFocal Loss
Handling extreme imbalanceModerateHigh
Training timeHighModerate
Risk of overfittingHighLow
Ease of implementationEasyModerate
Hyperparameter tuning complexityLowHigh
✅
Verdict
Default to focal loss for datasets with extreme imbalance and large size; switch to oversampling when the dataset is smaller and the minority class is not extremely rare.
💡
Key Insight
Focal loss excels in focusing on hard examples, making it ideal for extreme class imbalance in large datasets.
⚠️
Interview Trap
❌ Wrong: Always use oversampling because it increases minority class representation.
Ignoring the risk of overfitting and the computational cost of larger datasets.
✅ Consider the dataset size and class imbalance severity; focal loss is often better for extreme imbalance in large datasets.
Oversampling
"Increases the representation of minority classes, making them more visible to the model."
Weaknesses
  • Can lead to overfitting on the minority class due to repeated samples.
  • Increases training time as the dataset size grows.
  • May not handle extreme imbalance well if the minority class is very small.
Ideal When
  • When n < 100K and the minority class is at least 5% of the dataset.
  • When you have sufficient computational resources to handle increased data size.
  • When the model is prone to underfitting and can benefit from more data.
A financial institution using oversampling to detect fraudulent transactions where fraud cases are rare but critical.
Focal Loss
"Focuses learning on hard-to-classify examples, reducing the impact of easy negatives."
Weaknesses
  • Requires careful tuning of hyperparameters like gamma.
  • May not be as effective if the model architecture is not complex enough to leverage the loss function.
  • Can be computationally expensive due to the additional calculations per sample.
Ideal When
  • When n > 100K and the minority class is less than 1% of the dataset.
  • When using deep learning models with sufficient capacity to learn complex patterns.
  • When you have a limited budget for data augmentation or collection.
A tech company using focal loss in object detection models where certain objects are rare in images.
🐍
Oversampling - Example
▼ Show
Python
import numpy as np
from sklearn.utils import resample
X_minority = X[y == 1]
X_upsampled = resample(X_minority, replace=True, n_samples=len(X[y == 0]), random_state=42)
X_balanced = np.vstack((X[y == 0], X_upsampled))
y_balanced = np.hstack((y[y == 0], np.ones(len(X_upsampled))))
🐍
Focal Loss - Example
▼ Show
Python
import torch
import torch.nn as nn
def focal_loss(inputs, targets, alpha=0.25, gamma=2.0):
    BCE_loss = nn.BCEWithLogitsLoss(reduction='none')(inputs, targets)
    pt = torch.exp(-BCE_loss)
    F_loss = alpha * (1-pt)**gamma * BCE_loss
    return F_loss.mean()
✅
Use Oversampling When
  • •When n < 100K and minority class is >5% of the dataset.
  • •When you have ample computational resources for increased data size.
  • •When the model is simple and prone to underfitting.
✅
Use Focal Loss When
  • •When n > 100K and minority class is <1% of the dataset.
  • •When using complex models capable of leveraging advanced loss functions.
  • •When computational resources are limited for data augmentation.
↳
Nuances
  • Oversampling can lead to overfitting if the minority class is too small.
  • Focal loss requires careful tuning of gamma, which can be non-trivial.
  • In cases of extreme imbalance, focal loss often outperforms oversampling due to its focus on hard examples.
  • Oversampling is more straightforward to implement but may not scale well with very large datasets.

Practice

(1/5)
1. In the context of evaluating search engine results, which metric is most appropriate for measuring the quality of the top-ranked results?
easy
A. Mean Reciprocal Rank (MRR)
B. Mean Average Precision (MAP)
C. Normalized Discounted Cumulative Gain (NDCG)
D. Precision at K

Solution

  1. Step 1: Understand the focus of each metric

    MRR is specifically designed to evaluate the rank of the first relevant result, making it ideal for top-ranked results.

  2. Step 2: Eliminate wrong options

    MAP considers all relevant documents, NDCG focuses on the entire ranking, and Precision at K is not specific to the first relevant result.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    MRR focuses on the first relevant result. [OK]
Hint: MRR focuses on the first relevant result. [OK]
Common Mistakes:
2. In the context of why machine learning exists, which of the following problems is most suitable for a machine learning approach rather than a rule-based system?
easy
A. Predicting the next word in a sentence
B. Calculating the sum of a list of numbers
C. Sorting a list of integers
D. Checking if a number is prime

Solution

  1. Step 1: Identify the nature of the problem.

    Predicting the next word in a sentence involves understanding context and patterns, which are complex and not easily defined by rules.

  2. Step 2: Eliminate rule-based problems.

    Options A, C, and D are deterministic and can be solved with explicit algorithms.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    ML for patterns, rules for logic. [OK]
Hint: ML excels at pattern recognition, not deterministic logic. [OK]
Common Mistakes:
3. A CNN model is overfitting on the training data. Which of the following is a likely cause related to pooling layers?
medium
A. Pooling layers are too aggressive, losing important features
B. Pooling layers are not used, leading to high feature map dimensions
C. Pooling layers are applied after every convolutional layer
D. Pooling layers use a large stride, reducing model complexity

Solution

  1. Step 1: Analyze pooling impact

    Without pooling, feature maps remain large, increasing model complexity and risk of overfitting.

  2. Step 2: Eliminate wrong options

    Pooling layers are too aggressive, losing important features is incorrect because aggressive pooling would reduce overfitting. Pooling layers are applied after every convolutional layer is incorrect as regular pooling helps reduce overfitting. Pooling layers use a large stride, reducing model complexity is incorrect because large strides reduce complexity, not increase it.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    No pooling increases overfitting risk. [OK]
Hint: No pooling increases overfitting risk. [OK]
Common Mistakes:
4. When deciding between using a pretrained model or training from scratch for a specific task, which scenario favors training from scratch?
medium
A. The task involves a domain with no available pretrained models.
B. The task has a very large and diverse dataset available.
C. The task is similar to the one the pretrained model was trained on.
D. The task requires real-time inference with minimal latency.

Solution

  1. Step 1: Identify when pretraining is less beneficial.

    Pretraining is less useful when no relevant pretrained models exist.

  2. Step 2: Eliminate wrong options.

    The task is similar to the one the pretrained model was trained on. favors using pretrained models due to task similarity. The task has a very large and diverse dataset available. also favors pretrained models due to data availability. The task requires real-time inference with minimal latency. is unrelated to the choice between pretraining and training from scratch.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    No pretrained model? Train from scratch. [check]
Hint: No pretrained model? Train from scratch. [check]
Common Mistakes:
5. How would you adapt K-Means to handle a dataset with categorical features?
hard
A. Use one-hot encoding and apply standard K-Means
B. Use K-Modes clustering instead
C. Convert categorical features to numerical using label encoding
D. Ignore categorical features and cluster on numerical features only

Solution

  1. Step 1: Understand K-Means limitations

    K-Means is not suitable for categorical data as it relies on Euclidean distance.

  2. Step 2: Choose the right adaptation

    K-Modes is designed for clustering categorical data, making it the best choice.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    K-Modes for categorical data [OK]
Hint: K-Modes for categorical data [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard