Custom transformers in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Custom transformers

Which metric matters for Custom Transformers and WHY

Custom transformers change data before it goes into a model. The main goal is to improve the model's results. So, the key metrics to watch are the model's accuracy, precision, recall, and F1 score after using the transformer. These show if the data change helped the model learn better.

Also, check the consistency of the transformer: it should always transform data the same way. This ensures the model gets reliable input.

Confusion Matrix Example

Suppose a custom transformer improves a spam email detector. After training, the confusion matrix might look like this:

      | Predicted Spam | Predicted Not Spam |
      |----------------|--------------------|
      | True Positives (TP) = 90  | False Positives (FP) = 15 |
      | False Negatives (FN) = 10 | True Negatives (TN) = 85  |
    

Total emails = 90 + 10 + 15 + 85 = 200

From this, we calculate:

• Precision = TP / (TP + FP) = 90 / (90 + 15) = 0.857
• Recall = TP / (TP + FN) = 90 / (90 + 10) = 0.9
• F1 Score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.878

Precision vs Recall Tradeoff with Custom Transformers

Custom transformers can affect precision and recall differently. For example:

• If the transformer removes too many words, the model might miss spam emails (low recall).
• If it keeps too many irrelevant words, the model might wrongly mark good emails as spam (low precision).

Choosing the right transformer means balancing these. For spam detection, high precision is important to avoid losing good emails. For medical tests, high recall is key to catch all cases.

Good vs Bad Metric Values for Custom Transformers

Good:

• Precision and recall both above 0.8, showing balanced performance.
• F1 score close to or above 0.85, indicating good overall accuracy.
• Consistent transformation results on new data.

Bad:

• Precision or recall below 0.5, meaning many errors.
• F1 score below 0.6, showing poor balance.
• Transformer changes data unpredictably, causing model confusion.

Common Pitfalls in Metrics for Custom Transformers

• Accuracy paradox: High accuracy can be misleading if data is imbalanced. For example, if spam is rare, a model that always says "not spam" has high accuracy but is useless.
• Data leakage: If the transformer uses information from the test set during training, metrics will look better but won't work in real life.
• Overfitting: A transformer tuned too much on training data may not help on new data, causing metrics to drop.

Self-Check Question

Your model with a custom transformer has 98% accuracy but only 12% recall on fraud detection. Is it good for production?

Answer: No. The low recall means the model misses most fraud cases, which is dangerous. Even with high accuracy, the model fails its main job. You should improve recall before using it.

Key Result

Custom transformers should improve model precision, recall, and F1 score while ensuring consistent data transformation.

Practice

1. What is the main purpose of creating a custom transformer in machine learning pipelines?

easy

A. To train a machine learning model directly

B. To define a reusable data processing step with fit and transform methods

C. To visualize data distributions

D. To store the final predictions of a model

Solution

1. Step 1: Understand the role of transformers

   Transformers process data by learning parameters in fit and applying changes in transform.

2. Step 2: Identify the purpose of custom transformers

   Custom transformers let you create your own data processing steps reusable in pipelines.

3. Final Answer:

   To define a reusable data processing step with fit and transform methods -> Option B

4. Quick Check:

   Custom transformer = reusable data step [OK]

Hint: Custom transformers handle data prep, not model training [OK]

Common Mistakes:

• Confusing transformers with models
• Thinking transformers visualize data
• Assuming transformers store predictions

2. Which of the following is the correct way to start defining a custom transformer class in Python using scikit-learn?

easy

A. class MyTransformer(Pipeline):

B. class MyTransformer(Model):

C. class MyTransformer(BaseEstimator, TransformerMixin):

D. def MyTransformer():

Solution

1. Step 1: Recall inheritance for custom transformers

   Custom transformers inherit from BaseEstimator and TransformerMixin to get fit and transform methods.

2. Step 2: Match correct class definition syntax

   class MyTransformer(BaseEstimator, TransformerMixin): correctly shows class inheritance from BaseEstimator and TransformerMixin.

3. Final Answer:

   class MyTransformer(BaseEstimator, TransformerMixin): -> Option C

4. Quick Check:

   Inheritance from BaseEstimator and TransformerMixin = class MyTransformer(BaseEstimator, TransformerMixin): [OK]

Hint: Custom transformers inherit BaseEstimator and TransformerMixin [OK]

Common Mistakes:

• Using Model or Pipeline as base classes
• Defining transformer as a function
• Missing inheritance entirely

3. Given this custom transformer code snippet, what will print(transformed_data) output?

from sklearn.base import BaseEstimator, TransformerMixin
import numpy as np

class AddConstant(BaseEstimator, TransformerMixin):
    def __init__(self, constant=1):
        self.constant = constant
    def fit(self, X, y=None):
        return self
    def transform(self, X):
        return X + self.constant

X = np.array([[1, 2], [3, 4]])
transformer = AddConstant(constant=5)
transformed_data = transformer.fit_transform(X)
print(transformed_data)

medium

A. [[6 7]
[8 9]]

B. [[1 2]
[3 4]]

C. [[5 5]
[5 5]]

D. Error: fit_transform method not defined

Solution

1. Step 1: Understand transform method behavior

   The transform method adds the constant (5) to every element in X.

2. Step 2: Calculate transformed data

   Original X is [[1,2],[3,4]]. Adding 5 gives [[6,7],[8,9]].

3. Final Answer:

   [[6 7] [8 9]] -> Option A

4. Quick Check:

   Adding constant 5 to X = [[6 7] [8 9]] [OK]

Hint: transform adds constant to all elements [OK]

Common Mistakes:

• Thinking fit_transform is missing
• Forgetting to add constant
• Confusing output with original data

4. What is wrong with this custom transformer code?

from sklearn.base import BaseEstimator, TransformerMixin

class MultiplyTransformer(BaseEstimator, TransformerMixin):
    def __init__(self, factor=2):
        self.factor = factor
    def fit(self, X, y=None):
        return self
    def transform(self, X):
        return X * self.factor

transformer = MultiplyTransformer(factor=3)
result = transformer.transform([1, 2, 3])
print(result)

medium

A. transform method should convert input to numpy array before multiplying

B. fit method is missing a return statement

C. factor should be a list, not an int

D. Class should inherit from Pipeline, not BaseEstimator

Solution

1. Step 1: Check input type handling in transform

   Input is a list, multiplying list by int repeats list instead of element-wise multiply.

2. Step 2: Fix transform to convert input to numpy array

   Converting input to numpy array allows element-wise multiplication as intended.

3. Final Answer:

   transform method should convert input to numpy array before multiplying -> Option A

4. Quick Check:

   List * int repeats list, need numpy array for element-wise multiply [OK]

Hint: Use numpy arrays for element-wise math in transform [OK]

Common Mistakes:

• Assuming list * int does element-wise multiply
• Missing return in fit method (actually present)
• Wrong base class inheritance

5. You want to create a custom transformer that replaces missing values in a dataset with the median of each column, then scales the data by dividing by the max value per column. Which approach correctly combines these steps in one transformer?

hard

A. In fit, replace missing values; in transform, compute medians and max values

B. Use two separate transformers instead of one custom transformer

C. Only implement transform method to do all steps without fit

D. In fit, compute medians and max values; in transform, replace missing with medians and divide by max values

Solution

1. Step 1: Understand fit and transform roles

   fit calculates statistics (median, max) from training data; transform applies these to new data.

2. Step 2: Apply correct sequence in methods

   In fit, compute medians and max values; in transform, replace missing with medians and divide by max values correctly computes medians and max in fit, then replaces missing and scales in transform.

3. Final Answer:

   In fit, compute medians and max values; in transform, replace missing with medians and divide by max values -> Option D

4. Quick Check:

   fit learns stats, transform applies them [OK]

Hint: fit learns stats; transform applies them to data [OK]

Common Mistakes:

• Doing data replacement in fit instead of transform
• Skipping fit method
• Using separate transformers unnecessarily