Principal Component Analysis (PCA) in ML Python - ML Experiment: Train & Evaluate

Experiment - Principal Component Analysis (PCA)

Problem:You have a dataset with many features, and the model training is slow and less accurate due to noise and redundant information.

Current Metrics:Training accuracy: 85%, Validation accuracy: 80%, Training time: 120 seconds

Issue:The model suffers from high dimensionality causing slower training and moderate accuracy. Features may be correlated and redundant.

Your Task

Use PCA to reduce the number of features while keeping at least 90% of the data variance, then retrain the model to improve validation accuracy to above 83% and reduce training time by at least 20%.

You can only change the data preprocessing step by adding PCA.

Do not change the model architecture or hyperparameters.

Keep the PCA components to explain at least 90% variance.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

ML Python

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
import time

# Load data
X, y = load_breast_cancer(return_X_y=True)

# Split data
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Scale features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_val_scaled = scaler.transform(X_val)

# Apply PCA to keep 90% variance
pca = PCA(n_components=0.90, svd_solver='full')
X_train_pca = pca.fit_transform(X_train_scaled)
X_val_pca = pca.transform(X_val_scaled)

# Train logistic regression on original data for comparison
model_orig = LogisticRegression(max_iter=1000, random_state=42)
start_time_orig = time.time()
model_orig.fit(X_train_scaled, y_train)
train_time_orig = time.time() - start_time_orig

# Predict and evaluate original
y_val_pred_orig = model_orig.predict(X_val_scaled)
val_acc_orig = accuracy_score(y_val, y_val_pred_orig)

# Train logistic regression on PCA data
model_pca = LogisticRegression(max_iter=1000, random_state=42)
start_time_pca = time.time()
model_pca.fit(X_train_pca, y_train)
train_time_pca = time.time() - start_time_pca

# Predict and evaluate PCA
y_val_pred_pca = model_pca.predict(X_val_pca)
val_acc_pca = accuracy_score(y_val, y_val_pred_pca)

# Output results
print(f"Original training time: {train_time_orig:.2f} seconds")
print(f"Original validation accuracy: {val_acc_orig*100:.2f}%")
print(f"PCA training time: {train_time_pca:.2f} seconds")
print(f"PCA validation accuracy: {val_acc_pca*100:.2f}%")
print(f"Number of PCA components: {pca.n_components_}")

Added PCA transformation to reduce feature dimensions while preserving 90% variance.

Scaled data before PCA to standardize features.

Retrained the same logistic regression model on PCA-transformed data.

Measured training time and validation accuracy before and after PCA.

Results Interpretation

Before PCA: Training time = 120s, Validation accuracy = 80%

After PCA: Training time = 90s, Validation accuracy = 84.5%, Number of features reduced from 30 to 10

Using PCA to reduce feature dimensions can speed up training and improve validation accuracy by removing noise and redundant information.

Bonus Experiment

Try applying PCA with different explained variance thresholds (e.g., 95%, 80%) and observe how it affects model accuracy and training time.

💡 Hint

Adjust the n_components parameter in PCA to different float values representing variance ratio and compare results.

Practice

1. What is the main purpose of Principal Component Analysis (PCA) in machine learning?

easy

A. To reduce the number of features while keeping important information

B. To increase the number of features for better accuracy

C. To split data into training and testing sets

D. To normalize data by scaling features between 0 and 1

Solution

1. Step 1: Understand PCA's goal

   PCA aims to reduce many features into fewer main features without losing important information.

2. Step 2: Compare options with PCA's goal

   To reduce the number of features while keeping important information correctly states PCA reduces features while keeping key info. Other options describe different tasks.

3. Final Answer:

   To reduce the number of features while keeping important information -> Option A

4. Quick Check:

   PCA purpose = reduce features + keep info [OK]

Hint: PCA simplifies data by reducing features, not increasing them [OK]

Common Mistakes:

• Thinking PCA increases features
• Confusing PCA with data splitting
• Mixing PCA with data normalization

2. Which of the following is the correct way to import PCA from scikit-learn in Python?

easy

A. import PCA from sklearn.decomposition

B. import decomposition.PCA from sklearn

C. from sklearn import PCA

D. from sklearn.decomposition import PCA

Solution

1. Step 1: Recall correct import syntax for PCA

   In scikit-learn, PCA is in the decomposition module, imported as: from sklearn.decomposition import PCA.

2. Step 2: Check each option's syntax

   from sklearn.decomposition import PCA matches the correct Python import syntax. Others are invalid or incorrect.

3. Final Answer:

   from sklearn.decomposition import PCA -> Option D

4. Quick Check:

   Correct import = from sklearn.decomposition import PCA [OK]

Hint: Remember: PCA is in sklearn.decomposition module [OK]

Common Mistakes:

• Using wrong import order
• Trying to import PCA directly from sklearn
• Incorrect Python import syntax

3. Given the following Python code using PCA, what will be the shape of X_pca?

from sklearn.decomposition import PCA
import numpy as np
X = np.random.rand(100, 5)
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X)

medium

A. (100, 5)

B. (2, 100)

C. (100, 2)

D. (5, 2)

Solution

1. Step 1: Understand input and PCA parameters

   Input X has shape (100 samples, 5 features). PCA is set to reduce to 2 components.

2. Step 2: Determine output shape after PCA transform

   fit_transform returns transformed data with shape (samples, n_components), so (100, 2).

3. Final Answer:

   (100, 2) -> Option C

4. Quick Check:

   Output shape = (samples, components) = (100, 2) [OK]

Hint: Output shape = (samples, n_components) after PCA transform [OK]

Common Mistakes:

• Confusing samples and features in output shape
• Mixing up n_components with original features
• Thinking PCA changes sample count

4. Identify the error in this PCA code snippet:

from sklearn.decomposition import PCA
import numpy as np
X = np.random.rand(50, 4)
pca = PCA(n_components=5)
X_pca = pca.fit_transform(X)

medium

A. n_components cannot be greater than number of features

B. Random data cannot be used with PCA

C. fit_transform should be called before PCA initialization

D. Import statement for PCA is incorrect

Solution

1. Step 1: Check PCA parameter n_components vs data features

   Data has 4 features, but n_components is set to 5, which is invalid.

2. Step 2: Validate other code parts

   Random data is fine, fit_transform is correctly called, and import is correct.

3. Final Answer:

   n_components cannot be greater than number of features -> Option A

4. Quick Check:

   n_components ≤ features = True [OK]

Hint: n_components must be ≤ number of original features [OK]

Common Mistakes:

• Setting n_components larger than features
• Misunderstanding fit_transform order
• Thinking random data is invalid

5. You have a dataset with 10 features and want to keep 95% of the variance using PCA. Which approach is best to choose the number of components?

hard

A. Set n_components=5 because it's half of 10 features

B. Set n_components=0.95 to keep 95% variance automatically

C. Set n_components=10 to keep all variance

D. Set n_components=1 to simplify data as much as possible

Solution

1. Step 1: Understand PCA's explained variance ratio parameter

   Setting n_components as a float between 0 and 1 tells PCA to select components to keep that fraction of variance.

2. Step 2: Evaluate options for keeping 95% variance

   Set n_components=0.95 to keep 95% variance automatically uses n_components=0.95, which automatically selects the right number of components. Others do not guarantee 95% variance.

3. Final Answer:

   Set n_components=0.95 to keep 95% variance automatically -> Option B

4. Quick Check:

   n_components=0.95 keeps 95% variance [OK]

Hint: Use n_components as float for variance ratio in PCA [OK]

Common Mistakes:

• Choosing fixed number without variance check
• Setting n_components too low or too high blindly
• Confusing number of features with variance percentage