Bird
Raised Fist0

The Curse of Dimensionality - Why High Dimensions Hurt

THEORYintermediateInterview freq: medium
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🐛
The Feature Overload Dilemma
Started immediately after the last feature update, ongoing for 3 days.

A recommendation system for an e-commerce platform, processing 500K user interactions daily.

Symptom
Model accuracy is stable, but recall has dropped to 0. AUC decreased from 0.85 to 0.65 after adding new features.
📡
Signals
metric
Recall on the test set
Recall dropped to 0 after the last feature update.
metric
AUC on the test set
AUC decreased from 0.85 to 0.65 post-update.
feature_importance
Feature importance ranking
New features have high importance but low correlation with the target.
distribution
Distribution of new features
New features are sparse and have high dimensionality.
log
Model training logs
Training time increased significantly after adding new features.
🔬
Hypotheses
The model is suffering from the curse of dimensionality due to high-dimensional sparse features.
Supporting Signals
Recall dropped to 0AUC decreased from 0.85 to 0.65New features are sparse and have high dimensionality
Verify: Check the sparsity and dimensionality of the new features and their impact on model performance.
import numpy as np
sparsity = np.mean(np.sum(new_features == 0, axis=1) / new_features.shape[1])
print('Sparsity:', sparsity)
The new features are not informative and are causing overfitting.
Supporting Signals
New features have high importance but low correlation with the target
Verify: Analyze the correlation between new features and the target variable.
correlations = [np.corrcoef(new_features[:, i], target)[0, 1] for i in range(new_features.shape[1])]
print('Correlations:', correlations)
🎯
Root Cause
The addition of high-dimensional sparse features led to the curse of dimensionality, degrading model performance.
Why It Happened

New features were added without proper feature selection or dimensionality reduction, leading to sparsity and high dimensionality.

🔧
Fix Strategy
ImmediateRollback to the previous model version without the new features.
Short-termPerform feature selection to remove uninformative features and reduce dimensionality.
Long-termImplement a feature engineering pipeline that includes dimensionality reduction techniques like PCA.
🐍
Python Fix
Python
from sklearn.decomposition import PCA
pca = PCA(n_components=10)
reduced_features = pca.fit_transform(new_features)
model.fit(reduced_features, target)
✗
Common Wrong Diagnoses
▼ Show
The model is overfitting due to too many features.
Why candidates think this: High feature count often leads to overfitting.
Why wrong: Overfitting would typically show high training performance, but recall is 0.
The data pipeline introduced a bug affecting feature values.
Why candidates think this: A sudden drop in performance suggests data issues.
Why wrong: Feature values are correct; the issue is with their high dimensionality and sparsity.
📊
Monitoring to Add
  • Monitor recall and AUC metrics after each feature update.
  • Track feature sparsity and dimensionality metrics.
  • Set alerts for significant drops in model performance metrics.
📋
Process Change

Introduce a mandatory feature selection and validation step before deploying new features.

🏗️
Architectural Change

Design the system to automatically apply dimensionality reduction techniques to new features.

💡
Key Insight
High-dimensional sparse features can severely degrade model performance due to the curse of dimensionality.

Practice

(1/5)
1. 
import torch
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
for epoch in range(10):
    for data, target in dataloader:
        optimizer.zero_grad()
        output = model(data)
        loss = loss_fn(output, target)
        loss.backward()
        optimizer.step()
print(optimizer.param_groups[0]['lr'])
What will be printed after the loop?
easy
A. 0.0
B. 0.01
C. 0.1
D. Error

Solution

  1. Step 1: Understand the code.

    The learning rate is set to 0.1 and not changed in the loop.

  2. Step 2: Check for learning rate updates.

    SGD without a scheduler keeps the learning rate constant.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    SGD without scheduler keeps lr constant. [check]
Hint: SGD without scheduler keeps lr constant. [check]
Common Mistakes:
2. Given the following Python code snippet, what will be the output if the model is trained with biased data?
from sklearn.metrics import accuracy_score

y_true = [0, 1, 1, 0, 1]
y_pred = [0, 1, 1, 0, 0]

accuracy = accuracy_score(y_true, y_pred)
print(accuracy)
easy
A. 0.4
B. 0.8
C. 0.6
D. 0.2

Solution

  1. Step 1: Calculate the accuracy.

    Accuracy is the ratio of correctly predicted instances to the total instances. Here, 3 out of 5 predictions are correct.

  2. Step 2: Compute the accuracy.

    Correct predictions: [0, 1, 1] -> 0.6

  3. Final Answer:

    0.6 -> 0.6

  4. Quick Check:

    Count correct predictions, divide by total. [OK]
Hint: Count correct predictions, divide by total. [OK]
Common Mistakes:
3. When choosing between BPE and WordPiece for a language model, which scenario makes WordPiece a better choice?
medium
A. When the vocabulary size needs to be minimized.
B. When handling a language with a high degree of inflection.
C. When the model needs to handle rare words efficiently.
D. When the model is primarily used for text generation tasks.

Solution

  1. Step 1: Identify the strengths of WordPiece.

    WordPiece is designed to handle rare words efficiently by breaking them into subwords.

  2. Step 2: Eliminate wrong options.

    Minimizing vocabulary size (A) and handling inflection (B) are not specific strengths of WordPiece; text generation (D) is not a specific advantage.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    WordPiece excels with rare words. [OK]
Hint: WordPiece excels with rare words. [OK]
Common Mistakes:
4. In a real-time prediction system, how should you handle data splits to ensure both model performance and system efficiency?
hard
A. Use a single train-test split to minimize latency.
B. Use a static validation set to simplify the system.
C. Use a large test set to ensure model robustness.
D. Use a rolling window validation approach to adapt to new data.

Solution

  1. Step 1: Consider real-time constraints

    Real-time systems need to adapt to new data while maintaining performance.

  2. Step 2: Evaluate options for adaptability and efficiency

    Use a single train-test split to minimize latency. risks overfitting. Use a large test set to ensure model robustness. misallocates resources. Use a static validation set to simplify the system. lacks adaptability to new data.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Rolling window adapts to new data efficiently. [OK]
Hint: Rolling window adapts to new data efficiently. [OK]
Common Mistakes:
5. You need to train a model with a learning rate that decreases over time. Which approach would best suit this requirement?
hard
A. Use a constant learning rate with Adam
B. Switch to RMSProp with a fixed decay rate
C. Use AdaGrad, as it inherently decreases learning rate
D. Implement a learning rate scheduler with SGD

Solution

  1. Step 1: Identify the requirement.

    The requirement is for a learning rate that decreases over time.

  2. Step 2: Match the requirement to the solution.

    A learning rate scheduler with SGD allows for controlled decrease over time.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Scheduler with SGD for decreasing lr. [check]
Hint: Scheduler with SGD for decreasing lr. [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard