Custom transformers in ML Python - Model Pipeline Trace

Model Pipeline - Custom transformers

This pipeline shows how custom transformers help prepare data before training a model. Custom transformers let us add special steps to clean or change data in ways built-in tools don’t offer.

Data Flow - 5 Stages

1Raw data input

1000 rows x 5 columns→Load raw data with missing values and text columns→1000 rows x 5 columns

[{'age': 25, 'income': 50000, 'city': 'NY', 'score': 0.8, 'missing_feature': null}, ...]

↓

2Custom transformer: Fill missing values

1000 rows x 5 columns→Replace missing values with column mean→1000 rows x 5 columns

[{'age': 25, 'income': 50000, 'city': 'NY', 'score': 0.8, 'missing_feature': 0.5}, ...]

↓

3Custom transformer: Encode city names

1000 rows x 5 columns→Convert city names to numbers→1000 rows x 5 columns

[{'age': 25, 'income': 50000, 'city': 2, 'score': 0.8, 'missing_feature': 0.5}, ...]

↓

4Feature scaling

1000 rows x 5 columns→Scale numeric features to 0-1 range→1000 rows x 5 columns

[{'age': 0.25, 'income': 0.5, 'city': 2, 'score': 0.8, 'missing_feature': 0.5}, ...]

↓

5Model training

1000 rows x 5 columns→Train model on processed features→Model trained

Model learns to predict target from features

Training Trace - Epoch by Epoch

Loss
0.7 |****
0.6 |*** 
0.5 |**  
0.4 |*   
0.3 |*   
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.65	0.60	Model starts learning, loss is high, accuracy low
2	0.50	0.72	Loss decreases, accuracy improves
3	0.40	0.80	Model learns important patterns
4	0.35	0.85	Training converges, accuracy rising
5	0.30	0.88	Loss low, accuracy high, good fit

Prediction Trace - 5 Layers

Layer 1: Input sample

Layer 2: Fill missing values transformer

Layer 3: Encode city transformer

Layer 4: Feature scaling

Layer 5: Model prediction

Model Quiz - 3 Questions

What does the custom transformer for missing values do?

A Replaces missing values with the column mean

B Removes rows with missing values

C Converts missing values to zero

D Leaves missing values unchanged

Key Insight

Custom transformers let us add special data cleaning and feature changes that fit our needs. This helps the model learn better by giving it cleaner, more useful data.