Bird
Raised Fist0

Implement PCA - Covariance Matrix, Eigenvectors, Projection

CODINGadvancedInterview freq: high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
▶
Execution Trace
Step 1/4
Original Data Matrix (X)

Data matrix X with shape (n_samples, n_features) is initialized.

🎯 Understanding the structure of the input data matrix is crucial.
X = np.array([[2.5, 2.4], [0.5, 0.7], [2.2, 2.9], [1.9, 2.2], [3.1, 3.0], [2.3, 2.7], [2, 1.6], [1, 1.1], [1.5, 1.6], [1.1, 0.9]])
Covariance Matrix (Σ)

Covariance matrix Σ is computed, showing variance and covariance between features.

🎯 Covariance matrix is key to understanding feature relationships.
mean_vec = np.mean(X, axis=0)
cov_matrix = np.cov(X.T)
Eigenvectors & Eigenvalues

Eigenvectors and eigenvalues are calculated from the covariance matrix.

🎯 Eigenvectors determine the new feature space directions.
eig_vals, eig_vecs = np.linalg.eig(cov_matrix)
Projection Matrix

Data is projected onto the new feature space using selected eigenvectors.

🎯 Projection reduces dimensionality while preserving variance.
eig_pairs = [(np.abs(eig_vals[i]), eig_vecs[:, i]) for i in range(len(eig_vals))]
eig_pairs.sort(key=lambda x: x[0], reverse=True)
matrix_w = np.hstack((eig_pairs[0][1].reshape(2, 1), eig_pairs[1][1].reshape(2, 1)))
Y = X.dot(matrix_w)
📊
Diagram
Diagram
┌───────────────────────────────┐
│ Original Data Matrix (X)      │
│ ┌───────────────┐             │
│ │ x11 x12 ...  │             │
│ │ x21 x22 ...  │             │
│ └───────────────┘             │
│          │                    │
│          ▼                    │
│ Covariance Matrix (Σ)         │
│ ┌───────────────┐             │
│ │ σ11 σ12 ...  │             │
│ │ σ21 σ22 ...  │             │
│ └───────────────┘             │
│          │                    │
│          ▼                    │
│ Eigenvectors & Eigenvalues    │
│ ┌───────────────┐             │
│ │ v1  λ1       │             │
│ │ v2  λ2       │             │
│ └───────────────┘             │
│          │                    │
│          ▼                    │
│ Projected Data (Y)            │
│ ┌───────────────┐             │
│ │ y11 y12 ...  │             │
│ │ y21 y22 ...  │             │
│ └───────────────┘             │
└───────────────────────────────┘
Labels
  • Original Data Matrix (X): Input data
  • Covariance Matrix (Σ): Measures data variance
  • Eigenvectors & Eigenvalues: Principal components
  • Projected Data (Y): Transformed data
🔑
Key Moments
▲
Q: Why is the covariance matrix important in PCA?
A: The covariance matrix captures the variance and covariance between features, which is essential for identifying the principal components that maximize variance.
Q: How do eigenvectors relate to the new feature space in PCA?
A: Eigenvectors determine the directions of the new feature space, with each eigenvector representing a principal component.
Q: What is the significance of projecting data onto eigenvectors?
A: Projecting data onto eigenvectors reduces dimensionality while retaining the most significant variance, simplifying the dataset while preserving essential information.
Trace Summary

This diagram illustrates the PCA process, starting with the original data matrix, computing the covariance matrix to understand feature variance, and then finding eigenvectors and eigenvalues to identify principal components. The data is then projected onto these components, reducing dimensionality while preserving variance. Key steps include understanding the covariance matrix's role, calculating eigenvectors and eigenvalues, and projecting data onto the new feature space. This process is crucial for dimensionality reduction in machine learning.

Primary Focus

✓ Understanding PCA implementation through covariance matrix, eigenvectors, and projection

Practice

(1/5)
1. What does the following Python code output when applied to an image array img? 
import numpy as np

def flip_image(img):
    return np.flip(img, axis=1)

img = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
flipped_img = flip_image(img)
print(flipped_img)
easy
A. [[3, 2, 1], [6, 5, 4], [9, 8, 7]]
B. [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
C. [[7, 8, 9], [4, 5, 6], [1, 2, 3]]
D. [[9, 8, 7], [6, 5, 4], [3, 2, 1]]

Solution

  1. Step 1: Understand the flip operation

    The np.flip function flips the array along the specified axis. Here, axis=1 means flipping horizontally.

  2. Step 2: Apply the flip to the array

    Flipping horizontally changes the order of elements in each row.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Horizontal flip reverses row elements. [OK]
Hint: Horizontal flip reverses row elements. [OK]
Common Mistakes:
2. 
import numpy as np

def iou(box1, box2):
    x1 = max(box1[0], box2[0])
    y1 = max(box1[1], box2[1])
    x2 = min(box1[2], box2[2])
    y2 = min(box1[3], box2[3])
    intersection = max(0, x2 - x1) * max(0, y2 - y1)
    area1 = (box1[2] - box1[0]) * (box1[3] - box1[1])
    area2 = (box2[2] - box2[0]) * (box2[3] - box2[1])
    union = area1 + area2 - intersection
    return intersection / union

box1 = [1, 1, 3, 3]
box2 = [2, 2, 4, 4]
print(iou(box1, box2))
easy
A. 0.5
B. 0.25
C. 0.14285714285714285
D. 0.3333333333333333

Solution

  1. Step 1: Calculate intersection and union areas

    Intersection area is 1, union area is 7.

  2. Step 2: Compute IoU

    IoU = 1/7 = 0.14285714285714285.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Intersection over Union formula [OK]
Hint: Intersection over Union formula [OK]
Common Mistakes:
3. What is the time complexity of computing the scaled dot-product attention for a single query with n keys?
easy
A. O(n^2)
B. O(n log n)
C. O(n)
D. O(1)

Solution

  1. Step 1: Identify operations

    For a single query, the main operations are dot product and softmax, both O(n).

  2. Step 2: Eliminate incorrect complexities

    O(n^2) and O(n log n) are incorrect as they imply more complex operations than present.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Single query, n keys -> O(n) [OK]
Hint: Single query, n keys -> O(n) [OK]
Common Mistakes:
4. What does the following Python code output?
import numpy as np

def softmax(x):
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum(axis=0)

x = np.array([2.0, 1.0, 0.1])
print(softmax(x))
easy
A. [0.659, 0.242, 0.099]
B. [0.5, 0.3, 0.2]
C. [0.665, 0.244, 0.091]
D. [0.7, 0.2, 0.1]

Solution

  1. Step 1: Calculate softmax

    Compute the exponentials of the input array after subtracting the max value for numerical stability.

  2. Step 2: Normalize

    Divide each exponential by the sum of all exponentials to get probabilities.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Softmax outputs sum to 1 [OK]
Hint: Softmax outputs sum to 1 [OK]
Common Mistakes:
5. In the context of logistic regression, which approach is correct for handling a dataset with highly imbalanced classes?
medium
A. Use accuracy as the primary evaluation metric.
B. Apply SMOTE to balance the dataset.
C. Increase the learning rate to improve minority class prediction.
D. Use a linear activation function instead of sigmoid.

Solution

  1. Step 1: Identify the problem with imbalanced classes

    Imbalanced classes can lead to biased predictions towards the majority class.

  2. Step 2: Choose a method to address imbalance

    SMOTE (Synthetic Minority Over-sampling Technique) is effective for balancing datasets.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    SMOTE balances classes effectively [OK]
Hint: SMOTE balances classes effectively [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard