Bird
Raised Fist0

Dimensionality Reduction - PCA, t-SNE, UMAP, When to Use Which

THEORYintermediateInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
PCA vs t-SNE

For visualizing high-dimensional data in a 2D plot, would you choose PCA or t-SNE? Justify your choice.

📊
Comparison
DimensionPCAt-SNE
Computational EfficiencyHighLow
InterpretabilityHighLow
Ability to Capture Non-linear RelationshipsLowHigh
ScalabilityHighLow
Preservation of Global StructureHighLow
✅
Verdict
Default to PCA for large datasets where interpretability and computational efficiency are priorities. Switch to t-SNE for smaller datasets where capturing non-linear relationships is crucial for visualization.
💡
Key Insight
Choose PCA for efficiency and interpretability; choose t-SNE for capturing complex, non-linear structures in smaller datasets.
⚠️
Interview Trap
❌ Wrong: Always use t-SNE for visualization because it captures non-linear relationships.
Ignoring the computational cost and scalability issues of t-SNE for large datasets.
✅ Consider the dataset size and computational resources; use PCA for large datasets where interpretability and efficiency are needed, and t-SNE for smaller datasets with complex structures.
PCA
"Efficient linear dimensionality reduction with interpretability."
Weaknesses
  • Assumes linear relationships, which may not capture complex structures.
  • Sensitive to scaling of features, requiring preprocessing.
  • Less effective for non-Gaussian distributed data.
Ideal When
  • n > 100K and computational resources are limited.
  • Data is approximately Gaussian distributed.
  • Interpretability of components is crucial.
PCA is used by financial institutions for risk management to reduce dimensionality while maintaining interpretability.
t-SNE
"Captures complex, non-linear relationships for visualization."
Weaknesses
  • Computationally expensive, especially for large datasets.
  • Results can vary significantly with different hyperparameters.
  • Not suitable for preserving global data structure.
Ideal When
  • n < 10K and visualization is the primary goal.
  • Data has complex, non-linear relationships.
  • Exploratory data analysis where local structure is more important than global.
t-SNE is used by bioinformatics companies to visualize gene expression data, capturing complex patterns.
🐍
PCA - Example
▼ Show
Python
import numpy as np
from sklearn.decomposition import PCA

X = np.random.rand(1000, 50)
pca = PCA(n_components=2)
X_reduced = pca.fit_transform(X)
print(X_reduced.shape)
🐍
t-SNE - Example
▼ Show
Python
import numpy as np
from sklearn.manifold import TSNE

X = np.random.rand(1000, 50)
tsne = TSNE(n_components=2)
X_embedded = tsne.fit_transform(X)
print(X_embedded.shape)
✅
Use PCA When
  • •n > 100K and computational resources are limited.
  • •Data is approximately Gaussian distributed.
  • •Interpretability of components is crucial.
✅
Use t-SNE When
  • •n < 10K and visualization is the primary goal.
  • •Data has complex, non-linear relationships.
  • •Exploratory data analysis where local structure is more important than global.
↳
Nuances
  • PCA can be surprisingly effective for visualization if the data is well-scaled and approximately Gaussian.
  • t-SNE can produce misleading visualizations if hyperparameters are not carefully tuned.
  • For very large datasets, t-SNE's computational cost can be prohibitive, making PCA a more practical choice.

Practice

(1/5)
1. Which activation function is most likely to suffer from the vanishing gradient problem?
easy
A. ReLU
B. GELU
C. Sigmoid
D. Leaky ReLU

Solution

  1. Step 1: Identify the vanishing gradient issue.

    The vanishing gradient problem occurs when gradients become too small, hindering learning.

  2. Step 2: Evaluate each option.

    ReLU and Leaky ReLU do not suffer from this as they have non-zero gradients for positive inputs. GELU is similar to ReLU in behavior. Sigmoid squashes inputs to a small range, leading to vanishing gradients.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Sigmoid squashes, gradients vanish. [OK]
Hint: Sigmoid squashes, gradients vanish. [OK]
Common Mistakes:
2. In a scenario where computational resources are limited, which approach is more suitable for hyperparameter tuning and why?
medium
A. Use a train-validation split and skip the test set.
B. Use a single train-test split to save resources.
C. Use cross-validation to ensure robust hyperparameter tuning.
D. Use a larger test set to ensure model generalization.

Solution

  1. Step 1: Evaluate resource constraints

    Cross-validation is computationally expensive but provides robust tuning.

  2. Step 2: Eliminate less suitable options

    Use a single train-test split to save resources. risks overfitting. Use a train-validation split and skip the test set. skips the test set, risking unseen data performance. Use a larger test set to ensure model generalization. misallocates resources to testing rather than tuning.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Cross-validation ensures robust tuning despite cost. [OK]
Hint: Cross-validation ensures robust tuning despite cost. [OK]
Common Mistakes:
3. A Naive Bayes model is underperforming on a dataset. What is the most likely cause?
medium
A. The features are highly correlated.
B. The features are not normalized.
C. The model has too many parameters.
D. The dataset is too small.

Solution

  1. Step 1: Identify Potential Causes

    Naive Bayes assumes feature independence, so correlation can degrade performance.

  2. Step 2: Eliminate Other Causes

    The model has too many parameters. is incorrect because Naive Bayes is not parameter-heavy. The features are not normalized. is incorrect because normalization is not critical for Naive Bayes. The dataset is too small. is incorrect because Naive Bayes can work well with small datasets.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Correlated features degrade Naive Bayes. [[check]]
Hint: Correlated features degrade Naive Bayes. [[check]]
Common Mistakes:
4. A high-dimensional dataset is causing a model to perform poorly. The model's accuracy is unexpectedly low. What is the most likely root cause?
medium
A. The data sparsity is causing the model to generalize poorly.
B. The model is underfitting due to insufficient complexity.
C. The model is overfitting due to too many parameters.
D. The learning rate is too high, causing unstable training.

Solution

  1. Step 1: Identify the impact of high dimensions.

    High dimensions often lead to data sparsity, affecting generalization.

  2. Step 2: Eliminate wrong options.

    The model is underfitting due to insufficient complexity. is about model complexity, not data sparsity. The model is overfitting due to too many parameters. is about overfitting, not generalization. The learning rate is too high, causing unstable training. is about training stability, not generalization.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    High dimensions lead to sparsity, hurting generalization. [OK]
Hint: High dimensions lead to sparsity, hurting generalization. [OK]
Common Mistakes:
5. When choosing between BPE and WordPiece for a language model, which scenario makes WordPiece a better choice?
medium
A. When the vocabulary size needs to be minimized.
B. When handling a language with a high degree of inflection.
C. When the model needs to handle rare words efficiently.
D. When the model is primarily used for text generation tasks.

Solution

  1. Step 1: Identify the strengths of WordPiece.

    WordPiece is designed to handle rare words efficiently by breaking them into subwords.

  2. Step 2: Eliminate wrong options.

    Minimizing vocabulary size (A) and handling inflection (B) are not specific strengths of WordPiece; text generation (D) is not a specific advantage.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    WordPiece excels with rare words. [OK]
Hint: WordPiece excels with rare words. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard