Bird
Raised Fist0

Support Vector Machines - Margin, Kernel Trick, When to Use

THEORYadvancedInterview freq: medium
Choose your preparation mode6 modes available
🐛
The Kernel Collapse
Issue started after a feature update last Monday.

A fraud detection model using SVM with RBF kernel, processing 500K transactions per day.

Symptom
Model accuracy remains stable at 0.92, but recall has dropped to 0.0 for fraudulent transactions.
📡
Signals
metric
Recall for fraudulent transactions in production
Dropped from 0.85 to 0.0 after last Monday's deploy
feature_importance
Feature importance ranking for 'transaction_amount'
Dropped from rank #1 (0.25) to rank #10 (0.05) post-deploy
distribution
Distribution of 'transaction_amount' feature
Shifted significantly after last Monday's deploy
log
Kernel function logs
Increased number of support vectors after last Monday's deploy
metric
Validation AUC
Stable at 0.91, no change post-deploy
🔬
Hypotheses
The new feature update caused the kernel to misclassify due to a shift in feature distribution.
Distribution of 'transaction_amount' featureFeature importance ranking for 'transaction_amount'
Verify: Check if the feature distribution shift aligns with the drop in recall.
import matplotlib.pyplot as plt; plt.hist(df['transaction_amount'], bins=50); plt.show()
The increase in support vectors indicates overfitting to the new feature distribution.
Kernel function logsIncreased number of support vectors
Verify: Analyze the number of support vectors before and after the deploy.
print('Support vectors:', len(svm_model.support_))
🎯
Root Cause
The feature update caused a significant shift in the 'transaction_amount' distribution, leading the RBF kernel to misclassify due to overfitting.

The feature update was not properly validated for distribution shifts, causing the SVM to overfit to the new data.

🔧
Fix Strategy
ImmediateRollback to the previous model version before the feature update.
Short-termRe-evaluate the feature update and adjust the kernel parameters to handle the new distribution.
Long-termImplement a robust feature validation pipeline to detect distribution shifts before deployment.
🐍
Python Fix
Python
from sklearn.model_selection import train_test_split; X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42); svm_model.fit(X_train, y_train)
✗
Common Wrong Diagnoses
▼ Show
The model is underfitting due to insufficient data.
Why candidates think this: The drop in recall suggests poor model performance.
Why wrong: The accuracy remains stable, indicating the model is not underfitting.
The kernel function is incorrectly implemented.
Why candidates think this: The increase in support vectors suggests a kernel issue.
Why wrong: The kernel function is correct; the issue is with data distribution.
📊
Monitoring to Add
  • Real-time monitoring of feature distribution changes in production.
  • Alerts for significant drops in recall for key classes.
  • Monitoring the number of support vectors post-deploy.
📋
Process Change

Introduce a mandatory feature validation step in the deployment pipeline.

🏗️
Architectural Change

Adopt a more flexible model architecture that can adapt to feature distribution changes, such as ensemble methods.

💡
Key Insight
A significant feature distribution shift can cause SVMs to misclassify, especially with sensitive kernels like RBF.

Practice

(1/5)
1. In the context of handling class imbalance, which technique is specifically designed to adjust the loss function to focus more on hard-to-classify examples?
easy
A. Oversampling
B. Undersampling
C. Cost-sensitive Learning
D. Focal Loss

Solution

  1. Step 1: Identify the technique that modifies the loss function.

    Focal Loss is designed to focus more on hard-to-classify examples by adjusting the loss function.
  2. Step 2: Eliminate wrong options.

    Oversampling and undersampling adjust the dataset, not the loss function. Cost-sensitive learning modifies the cost, not the focus on hard examples.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Focal Loss focuses on hard examples. [OK]
Hint: Focal Loss focuses on hard examples. [OK]
Common Mistakes:
2. Given the following Python code snippet, what will be the output?
import numpy as np
business_goals = np.array(['increase sales', 'reduce churn', 'improve satisfaction'])
ml_problems = np.array(['classification', 'regression', 'clustering'])
result = np.char.add(business_goals, ml_problems)
print(result)
easy
A. ['increase salesclassification', 'reduce churnregression', 'improve satisfactionclustering']
B. ['increase sales', 'reduce churn', 'improve satisfaction']
C. ['classification', 'regression', 'clustering']
D. Throws an error due to incompatible types

Solution

  1. Step 1: Understand the operation

    The np.char.add function concatenates corresponding elements of two arrays element-wise.
  2. Step 2: Eliminate wrong options

    ['increase sales', 'reduce churn', 'improve satisfaction'] and C are incorrect because they ignore the concatenation operation. Throws an error due to incompatible types is incorrect because np.char.add handles string arrays correctly.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    np.char.add concatenates strings element-wise. [OK]
Hint: np.char.add concatenates strings element-wise. [OK]
Common Mistakes:
3. A model performs well overall but poorly on a specific slice of data. What is the most likely cause?
medium
A. The slice has a different distribution than the training data.
B. The model has too many parameters.
C. The model is overfitting to the training data.
D. The learning rate is too high.

Solution

  1. Step 1: Analyze the problem

    Poor performance on a specific slice often indicates distribution mismatch.
  2. Step 2: Eliminate wrong options

    The model is overfitting to the training data. is incorrect as overfitting would affect overall performance. The model has too many parameters. is unrelated to slice-specific issues. The learning rate is too high. affects convergence, not slice performance.
  3. Final Answer:

    Option A -> Option A
  4. Quick Check:

    Slice mismatch = distribution issue. [OK]
Hint: Slice mismatch = distribution issue. [OK]
Common Mistakes:
4. A model is showing unexpectedly high validation accuracy but low test accuracy. What is the most likely cause?
medium
A. The model is underfitting the training data.
B. The test set is too similar to the training set.
C. The model has too many parameters.
D. The validation set is too small and not representative.

Solution

  1. Step 1: Diagnose the issue

    High validation accuracy but low test accuracy suggests the validation set may not be representative.
  2. Step 2: Eliminate wrong options

    The model is underfitting the training data. would affect both validation and test accuracy, The test set is too similar to the training set. would not cause low test accuracy, and The model has too many parameters. would affect both validation and test accuracy similarly.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    Small validation set = misleading accuracy [OK]
Hint: Small validation set = misleading accuracy [OK]
Common Mistakes:
5. A Lasso regression model is underperforming with high bias. What is the most likely cause?
medium
A. The features are not normalized, affecting the coefficients
B. The dataset is too large, leading to overfitting
C. The model complexity is too high, causing variance
D. The alpha parameter is too high, causing excessive shrinkage

Solution

  1. Step 1: Identify the effect of high alpha in Lasso.

    High alpha increases regularization strength, leading to high bias.
  2. Step 2: Eliminate wrong options.

    Large datasets do not cause overfitting. High complexity increases variance, not bias. Non-normalized features affect scale, not bias directly.
  3. Final Answer:

    Option D -> Option D
  4. Quick Check:

    High alpha = high bias in Lasso. [OK]
Hint: High alpha = high bias in Lasso. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard