Bird
Raised Fist0

Dimensionality Reduction - PCA, t-SNE, UMAP, When to Use Which

THEORYintermediateInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Dimensionality Reduction - PCA, t-SNE, UMAP, When to Use Which
Choosing the right dimensionality reduction technique for your data
theoryintermediateFreq: highClassical ML
Prerequisites:Curse of Dimensionality
💡
Why It Exists

High-dimensional data often leads to overfitting and computational inefficiency. Dimensionality reduction techniques like PCA, t-SNE, and UMAP help by reducing the number of features while preserving essential patterns, making models more robust and faster to train.

📖
What It Is

Dimensionality reduction is the process of reducing the number of random variables under consideration by obtaining a set of principal variables. Imagine compressing a large, detailed map into a simpler version that still shows the main roads and landmarks.

∑
Math Intuition
PCA finds the directions (principal components) that maximize variance, effectively capturing the most 'spread out' aspects of the data. t-SNE minimizes the divergence between probability distributions of data points in high and low dimensions, focusing on local structure. UMAP uses a topological approach to maintain both local and global data relationships.
⚙️
Core Mechanism

These techniques transform high-dimensional data into a lower-dimensional space while preserving important structures.

📋
Step by Step
  1. PCA: Compute the covariance matrix of the data.
  2. PCA: Calculate eigenvectors and eigenvalues of the covariance matrix.
  3. PCA: Project data onto the top k eigenvectors.
  4. t-SNE: Compute pairwise similarities in high-dimensional space.
  5. t-SNE: Optimize a cost function to match these similarities in low-dimensional space.
  6. UMAP: Construct a high-dimensional graph of the data.
  7. UMAP: Optimize a low-dimensional graph to be structurally similar.
🎯
Intuition
Think of PCA as finding the best-fitting line or plane through your data, t-SNE as arranging data points on a 2D canvas to reflect their similarities, and UMAP as creating a map that preserves both local and global data structures.
🐍
Python Example
Python
import numpy as np
from sklearn.decomposition import PCA

# Sample data
X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])

# Apply PCA
pca = PCA(n_components=1)
X_reduced = pca.fit_transform(X)
print(X_reduced)  # Output: [[-3.], [-1.], [ 1.], [ 3.]]
✅
When It Works
  • PCA is effective when the data is linearly correlated and you need a fast, interpretable reduction.
  • t-SNE excels in visualizing complex, non-linear relationships in small to medium-sized datasets.
  • UMAP is suitable for large datasets where preserving both local and global structures is crucial.
⚠️
When It Breaks
  • PCA fails with non-linear data structures, as it only captures linear relationships.
  • t-SNE struggles with very large datasets due to its computational complexity, leading to long runtimes.
  • UMAP can produce misleading results if the hyperparameters are not carefully tuned, especially in sparse data.
⇄
vs Autoencoders
Use This When
Use PCA, t-SNE, or UMAP when interpretability and visualization are priorities.
Use Alternative When
Use autoencoders when you have a large dataset and need a non-linear reduction that can be integrated into a neural network.
🎯
Interview Angles
  • Interviewer asks you to choose between PCA and t-SNE for a dataset with non-linear patterns → testing your understanding of linear vs. non-linear dimensionality reduction.
  • Interviewer presents a large dataset and asks which dimensionality reduction technique to use → assessing your knowledge of computational efficiency and scalability.
  • Interviewer asks how you would visualize high-dimensional data → evaluating your ability to select appropriate techniques for data exploration.
⚠️
Common Mistakes
Believing PCA can handle non-linear data well, leading to poor feature representation.
Assuming t-SNE is always better than PCA, ignoring its computational cost and scalability issues.
Thinking UMAP is a direct replacement for t-SNE without understanding its different assumptions and hyperparameter sensitivities.
📌
Summary
  • PCA is best for linear relationships and quick, interpretable reductions.
  • t-SNE is ideal for visualizing complex, non-linear structures in smaller datasets.
  • UMAP balances local and global structure preservation, suitable for large datasets.

Practice

(1/5)
1. Given the following PyTorch code snippet, what will be the output shape of the tensor after applying dropout during training?
import torch
import torch.nn as nn
x = torch.ones(5, 5)
dropout = nn.Dropout(p=0.5)
output = dropout(x)
easy
A. (5, 5) with all elements set to zero
B. (5, 5) with all elements halved
C. (5, 5) with some elements set to zero
D. (5, 5) with some elements doubled

Solution

  1. Step 1: Understand dropout effect

    Dropout randomly sets some elements to zero during training, while scaling the remaining elements.

  2. Step 2: Eliminate wrong options

    (5, 5) with all elements halved is incorrect because not all elements are halved; only non-zero elements are scaled. (5, 5) with all elements set to zero is incorrect because not all elements are set to zero. (5, 5) with some elements doubled is incorrect because elements are not doubled.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Dropout zeros some elements. [OK]
Hint: Dropout zeros some elements. [OK]
Common Mistakes:
2. A model trained with K-Fold cross-validation is underperforming on unseen data. What is a likely cause?
medium
A. The model is overfitting due to too many folds.
B. Data leakage between training and validation sets.
C. The model is underfitting due to too few folds.
D. The model is not complex enough for the data.

Solution

  1. Step 1: Consider data leakage as a common issue.

    Data leakage can occur if the same data points appear in both training and validation sets.

  2. Step 2: Eliminate wrong options.

    Too many folds (A) typically doesn't cause overfitting, too few folds (C) would not cause underperformance on unseen data, model complexity (D) is unrelated to cross-validation setup.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Check for data leakage in CV. [OK]
Hint: Check for data leakage in CV. [OK]
Common Mistakes:
3. When debugging a model with high variance, which approach is more suitable?
medium
A. Increase model complexity
B. Add more training data
C. Reduce regularization
D. Use a smaller model

Solution

  1. Step 1: Identify high variance symptoms

    High variance indicates overfitting, where the model performs well on training data but poorly on unseen data.

  2. Step 2: Choose the right approach

    Adding more training data helps the model generalize better, reducing overfitting.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    High variance = more data [OK]
Hint: High variance = more data [OK]
Common Mistakes:
4. In a system where data distribution shifts gradually over time, how should you design the learning process?
hard
A. Use continual learning without any additional mechanisms.
B. Use periodic retraining with a fixed schedule regardless of data shifts.
C. Use continual learning with a mechanism to detect and adapt to shifts.
D. Use periodic retraining only when performance drops significantly.

Solution

  1. Step 1: Consider the nature of gradual shifts.

    Gradual shifts require a system that can adapt over time without waiting for performance to drop significantly.

  2. Step 2: Evaluate the options.

    Use periodic retraining with a fixed schedule regardless of data shifts. is incorrect because a fixed schedule may not align with shifts. Use continual learning without any additional mechanisms. is incorrect because continual learning alone may not handle shifts effectively. Use periodic retraining only when performance drops significantly. is incorrect because waiting for performance drops can lead to suboptimal results.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Adapt to shifts with detection. [OK]
Hint: Adapt to shifts with detection. [OK]
Common Mistakes:
5. You are tasked with designing a system to detect fraudulent transactions. Which approach is most appropriate given the need for adaptability to new fraud patterns?
hard
A. Use a rule-based system with predefined fraud patterns.
B. Rely on manual inspection of transactions.
C. Use a static decision tree with fixed thresholds.
D. Implement a machine learning model that can be retrained with new data.

Solution

  1. Step 1: Consider the need for adaptability.

    Fraud patterns change over time, requiring a system that can learn and adapt.

  2. Step 2: Evaluate the options for adaptability.

    Only a machine learning model that can be retrained offers the necessary adaptability.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    ML adapts, rules are static. [OK]
Hint: ML adapts, rules are static. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard