Bird
Raised Fist0

Handling Class Imbalance - Oversampling, Undersampling, Focal Loss

THEORYintermediateInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Handling Class Imbalance - Oversampling, Undersampling, Focal Loss
Techniques to tackle class imbalance in classification tasks.
theoryintermediateFreq: very_highData & Features
Prerequisites:Classification Metrics - Precision, Recall, F1
💡
Why It Exists

Class imbalance occurs when one class significantly outnumbers others, leading models to be biased towards the majority class. For example, in fraud detection, the number of fraudulent transactions is much smaller than legitimate ones, causing models to predict 'not fraud' most of the time, missing actual fraud cases.

📖
What It Is

Handling class imbalance involves techniques to adjust the training data or loss function to ensure the model learns equally from all classes. Think of it as leveling the playing field so that minority classes have a fair chance to influence the model's predictions.

⚙️
Core Mechanism

These techniques either modify the dataset or the loss function to counteract the imbalance.

📋
Step by Step
  1. Oversampling: Duplicate instances of the minority class to balance the class distribution.
  2. Undersampling: Remove instances from the majority class to balance the class distribution.
  3. Focal Loss: Modify the loss function to focus more on hard-to-classify examples, reducing the impact of easy-to-classify majority class examples.
🎯
Intuition
Imagine a seesaw with a heavy weight on one side (majority class) and a light weight on the other (minority class). Oversampling adds more weight to the light side, undersampling removes weight from the heavy side, and focal loss adjusts the pivot point to balance the seesaw.
🐍
Python Example
Python
import numpy as np
from sklearn.utils import resample

# Example data
X = np.array([[1], [2], [3], [4], [5], [6]])
y = np.array([0, 0, 0, 1, 1, 1])

# Oversample minority class
X_resampled, y_resampled = resample(X[y == 1], y[y == 1],
                                    replace=True,
                                    n_samples=3,
                                    random_state=42)

# Combine with majority class
X_balanced = np.vstack((X[y == 0], X_resampled))
y_balanced = np.hstack((y[y == 0], y_resampled))

# Output: X_balanced.shape, y_balanced.shape
# Output: (6, 1), (6,)
✅
When It Works
  • When the class imbalance ratio is moderate (e.g., 1:5 to 1:10) and you want to ensure the model learns from minority classes.
  • When the cost of misclassifying the minority class is high, such as in fraud detection or medical diagnosis.
  • When you have enough data to apply oversampling without overfitting the model.
  • When you want to improve recall without significantly sacrificing precision.
⚠️
When It Breaks
  • When class imbalance is extreme (e.g., 1:1000), oversampling can lead to overfitting on the minority class.
  • When the dataset is small, undersampling can remove too much information, leading to poor generalization.
  • When the minority class is noisy, oversampling can amplify noise, leading to worse model performance.
  • When using focal loss, if the tuning of the focusing parameter is incorrect, it can lead to suboptimal model performance.
⇄
vs Class Weighting
Use This When
Use oversampling, undersampling, or focal loss when you want to directly manipulate the dataset or loss function to address imbalance.
Use Alternative When
Use class weighting when you prefer to adjust the importance of each class in the loss function without altering the dataset.
🎯
Interview Angles
  • Interviewer asks: 'How would you handle a dataset with a 1:100 class imbalance?' → testing your understanding of different techniques and their trade-offs.
  • Interviewer presents a scenario where a model has high accuracy but low recall on the minority class → assessing your ability to diagnose class imbalance issues.
  • Interviewer asks you to implement a solution for class imbalance in a coding exercise → evaluating your practical skills in applying oversampling or focal loss.
⚠️
Common Mistakes
Believing oversampling always improves model performance, leading to overfitting on the minority class.
Assuming undersampling is safe for small datasets, which can result in losing critical information.
Thinking focal loss is a one-size-fits-all solution without tuning its parameters, leading to suboptimal results.
Ignoring the impact of noise in the minority class when applying oversampling, which can degrade model performance.
📌
Summary
  • Handling class imbalance involves techniques like oversampling, undersampling, and focal loss to ensure balanced learning.
  • Each technique has specific conditions where it excels and potential pitfalls if misapplied.
  • Understanding the trade-offs and failure modes of these techniques is crucial for effective model training.

Practice

(1/5)
1. Given the following PyTorch code snippet, what will be the output shape of the model's output after fine-tuning?
import torch
import torch.nn as nn

class SimpleModel(nn.Module):
    def __init__(self):
        super(SimpleModel, self).__init__()
        self.features = nn.Sequential(
            nn.Linear(512, 256),
            nn.ReLU(),
            nn.Linear(256, 128)
        )
        self.classifier = nn.Linear(128, 10)

    def forward(self, x):
        x = self.features(x)
        x = self.classifier(x)
        return x

model = SimpleModel()
input_data = torch.randn(32, 512)
output = model(input_data)
print(output.shape)
easy
A. torch.Size([32, 512])
B. torch.Size([32, 128])
C. torch.Size([32, 10])
D. torch.Size([32, 256])

Solution

  1. Step 1: Trace the forward pass.

    The input shape is (32, 512). After passing through the features, it becomes (32, 128). The classifier then outputs (32, 10).

  2. Step 2: Eliminate wrong options.

    torch.Size([32, 512]) is incorrect because the output is not the same as the input. torch.Size([32, 256]) and C are intermediate shapes, not the final output.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Final layer determines output shape. [check]
Hint: Final layer determines output shape. [check]
Common Mistakes:
2. A model using wrapper methods for feature selection is overfitting. What is the most likely cause?
medium
A. Data leakage during feature selection
B. Inappropriate scoring function
C. Too few features selected
D. High variance in the dataset

Solution

  1. Step 1: Identify overfitting causes.

    Data leakage during feature selection can lead to overfitting as the model sees information from the validation set.

  2. Step 2: Eliminate wrong options.

    Too few features usually lead to underfitting. An inappropriate scoring function affects feature ranking, not directly overfitting. High variance can cause overfitting but is not specific to wrapper methods.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Data leakage = overfitting risk. [OK]
Hint: Data leakage causes overfitting. [OK]
Common Mistakes:
3. For a large streaming dataset where you need to maintain a representative sample, which sampling strategy is most appropriate and why?
medium
A. Stratified Sampling to ensure class balance.
B. Reservoir Sampling to handle streaming data efficiently.
C. Random Sampling because it's simple and fast.
D. Weighted Sampling to prioritize important data.

Solution

  1. Step 1: Identify the need for streaming data handling.

    Streaming data requires a strategy that can update samples dynamically.

  2. Step 2: Eliminate non-streaming strategies.

    Random (A) and Stratified (B) don't handle streaming efficiently. Weighted (D) is for prioritization, not streaming.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Reservoir = Streaming data handling [[check]]
Hint: Reservoir = Streaming data handling [[check]]
Common Mistakes:
4. In a fraud detection system, you need to balance precision and recall. Which metric would you choose and why?
hard
A. F1 Score, because it balances both precision and recall.
B. Precision, because false positives are more costly.
C. Recall, because missing a fraud case is more critical.
D. Accuracy, because it gives an overall performance measure.

Solution

  1. Step 1: Understand the need for balance.

    Fraud detection requires a balance between catching frauds and minimizing false alarms.

  2. Step 2: Choose the metric that balances precision and recall.

    F1 Score is the harmonic mean of precision and recall, providing a balance.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    F1 Score balances precision and recall. [OK]
Hint: F1 Score balances precision and recall. [OK]
Common Mistakes:
5. In a recommendation system, how would you apply feature engineering to improve the model's ability to capture user-item interactions?
hard
A. Create interaction terms between user and item features.
B. Apply one-hot encoding to user and item IDs.
C. Use collaborative filtering to generate features.
D. Normalize user and item features separately.

Solution

  1. Step 1: Identify the need for capturing interactions.

    Recommendation systems benefit from understanding user-item interactions.

  2. Step 2: Evaluate feature engineering techniques.

    Collaborative filtering (A) is a model, not feature engineering. One-hot encoding (B) doesn't capture interactions. Normalizing (D) doesn't enhance interaction capture. Interaction terms (C) explicitly model user-item interactions, improving the system's ability to recommend.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Interaction terms enhance user-item interaction capture. [OK]
Hint: Interaction terms enhance user-item interaction capture. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard