Feature union in ML Python - Model Pipeline Trace

Model Pipeline - Feature union

This pipeline combines different sets of features from the same data to help the model learn better. It joins features side-by-side so the model sees more information at once.

Data Flow - 5 Stages

1Raw data input

1000 rows x 5 columns→Collect original features→1000 rows x 5 columns

[[5.1, 3.5, 1.4, 0.2, 0], [4.9, 3.0, 1.4, 0.2, 0], ...]

↓

2Feature extraction A

1000 rows x 5 columns→Extract numeric features (e.g., mean, max)→1000 rows x 3 columns

[[3.05, 5.1, 1.4], [2.7, 4.9, 1.4], ...]

↓

3Feature extraction B

1000 rows x 5 columns→Extract categorical features (e.g., one-hot encoding)→1000 rows x 4 columns

[[1, 0, 0, 0], [1, 0, 0, 0], ...]

↓

4Feature union

Two sets: 1000 rows x 3 columns and 1000 rows x 4 columns→Combine features side-by-side→1000 rows x 7 columns

[[3.05, 5.1, 1.4, 1, 0, 0, 0], [2.7, 4.9, 1.4, 1, 0, 0, 0], ...]

↓

5Model training

1000 rows x 7 columns→Train classifier on combined features→Trained model

Model learns to predict target labels

Training Trace - Epoch by Epoch

Loss
0.7 |****
0.6 |*** 
0.5 |**  
0.4 |*   
0.3 |    
0.2 |    
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.65	0.60	Model starts learning with moderate loss and accuracy
2	0.48	0.75	Loss decreases and accuracy improves as model learns
3	0.35	0.85	Model shows good learning progress
4	0.28	0.90	Loss continues to decrease, accuracy rises
5	0.22	0.93	Model converges with low loss and high accuracy

Prediction Trace - 5 Layers

Layer 1: Input sample

Layer 2: Feature extraction A

Layer 3: Feature extraction B

Layer 4: Feature union

Layer 5: Model prediction

Model Quiz - 3 Questions

What does the feature union step do in the pipeline?

A Removes duplicate features

B Combines different feature sets side-by-side

C Splits data into training and test sets

D Normalizes all features to the same scale

Key Insight

Feature union helps the model by combining different types of features into one set, giving it more information to learn from. This usually improves accuracy and helps the model understand the data better.

Practice

1. What is the main purpose of using FeatureUnion in machine learning?

easy

A. To combine multiple feature extraction methods into a single feature set

B. To split data into training and testing sets

C. To reduce the number of features by selecting the best ones

D. To train multiple models and average their predictions

Solution

1. Step 1: Understand FeatureUnion's role

   FeatureUnion is used to combine different feature extraction methods so their outputs join into one feature set.

2. Step 2: Compare with other options

   Splitting data, feature selection, and model averaging are different tasks not done by FeatureUnion.

3. Final Answer:

   To combine multiple feature extraction methods into a single feature set -> Option A

4. Quick Check:

   FeatureUnion = Combine features [OK]

Hint: FeatureUnion joins features, not data splits or models [OK]

Common Mistakes:

• Confusing FeatureUnion with data splitting
• Thinking it selects features instead of combining
• Mixing it up with model ensemble methods

2. Which of the following is the correct way to create a FeatureUnion with two transformers named 'tf1' and 'tf2'?

easy

A. FeatureUnion(tf1=transformer1, tf2=transformer2)

B. FeatureUnion({'tf1': transformer1, 'tf2': transformer2})

C. FeatureUnion([('tf1', transformer1), ('tf2', transformer2)])

D. FeatureUnion(transformer1, transformer2)

Solution

1. Step 1: Recall FeatureUnion syntax

   FeatureUnion expects a list of tuples, each tuple with a name and a transformer.

2. Step 2: Check each option

   FeatureUnion([('tf1', transformer1), ('tf2', transformer2)]) uses a list of tuples correctly. Options B, C, and D use wrong data structures or missing list.

3. Final Answer:

   FeatureUnion([('tf1', transformer1), ('tf2', transformer2)]) -> Option C

4. Quick Check:

   FeatureUnion needs list of (name, transformer) tuples [OK]

Hint: Use list of (name, transformer) tuples for FeatureUnion [OK]

Common Mistakes:

• Passing a dictionary instead of list of tuples
• Passing transformers without names
• Passing transformers as separate arguments

3. Given the code below, what will be the shape of X_transformed?

from sklearn.pipeline import FeatureUnion
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import numpy as np

X = np.array([[1, 2, 3], [4, 5, 6]])

union = FeatureUnion([
    ('scale', StandardScaler()),
    ('pca', PCA(n_components=1))
])

X_transformed = union.fit_transform(X)

medium

A. (2, 1)

B. (2, 3)

C. (2, 2)

D. (2, 4)

Solution

1. Step 1: Analyze each transformer output

   StandardScaler keeps original shape (2 samples, 3 features) so output shape is (2,3). PCA with n_components=1 outputs (2,1).

2. Step 2: Combine outputs with FeatureUnion

   FeatureUnion concatenates outputs horizontally: (2,3) + (2,1) = (2,4).

3. Final Answer:

   (2, 4) -> Option D

4. Quick Check:

   Concatenate (2,3) and (2,1) = (2,4) [OK]

Hint: FeatureUnion concatenates horizontally, sum feature counts [OK]

Common Mistakes:

• Assuming PCA output replaces original features
• Thinking FeatureUnion stacks vertically
• Ignoring output shapes of individual transformers

4. You wrote this code but get an error:

union = FeatureUnion([
    ('scale', StandardScaler()),
    ('pca', PCA(n_components=3))
])

X_transformed = union.fit_transform([[1, 2], [3, 4], [5, 6]])

What is the likely cause of the error?

medium

A. PCA cannot have n_components greater than input features

B. StandardScaler requires 3D input, but input is 2D

C. FeatureUnion requires transformers to have fit_predict method

D. Input data must be a pandas DataFrame, not a list

Solution

1. Step 1: Check input data shape

   The input X = [[1,2],[3,4],[5,6]] has shape (3, 2), meaning 2 features.

2. Step 2: Analyze PCA configuration

   PCA(n_components=3) requests 3 components, but only 2 features are available, causing a ValueError.

3. Final Answer:

   PCA cannot have n_components greater than input features -> Option A

4. Quick Check:

   PCA n_components ≤ features [OK]

Hint: Check PCA n_components ≤ number of features [OK]

Common Mistakes:

• Assuming StandardScaler needs 3D input
• Thinking FeatureUnion needs fit_predict
• Believing input must be DataFrame

5. You want to combine text and numeric features for a model. You have a TfidfVectorizer for text and StandardScaler for numeric data. How do you use FeatureUnion to prepare the data correctly?

hard

A. Apply TfidfVectorizer and StandardScaler separately, then add their outputs manually

B. Use FeatureUnion with transformers for text and numeric, each applied to their columns via ColumnTransformer

C. Use FeatureUnion directly on raw data without preprocessing

D. Use StandardScaler on text data and TfidfVectorizer on numeric data

Solution

1. Step 1: Understand data types and transformers

   Text and numeric data need different preprocessing. TfidfVectorizer works on text, StandardScaler on numeric features.

2. Step 2: Use ColumnTransformer with FeatureUnion

   Apply each transformer to correct columns using ColumnTransformer, then combine with FeatureUnion to merge features.

3. Final Answer:

   Use FeatureUnion with transformers for text and numeric, each applied to their columns via ColumnTransformer -> Option B

4. Quick Check:

   Separate preprocessing per data type, then combine [OK]

Hint: Preprocess each data type separately, then combine features [OK]

Common Mistakes:

• Applying wrong transformer to wrong data type
• Skipping column selection before FeatureUnion
• Trying to combine raw data without preprocessing