Custom transformers in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Custom transformers

Which metric matters for Custom Transformers and WHY

Custom transformers change data before it goes into a model. The main goal is to improve the model's results. So, the key metrics to watch are the model's accuracy, precision, recall, and F1 score after using the transformer. These show if the data change helped the model learn better.

Also, check the consistency of the transformer: it should always transform data the same way. This ensures the model gets reliable input.

Confusion Matrix Example

Suppose a custom transformer improves a spam email detector. After training, the confusion matrix might look like this:

      | Predicted Spam | Predicted Not Spam |
      |----------------|--------------------|
      | True Positives (TP) = 90  | False Positives (FP) = 15 |
      | False Negatives (FN) = 10 | True Negatives (TN) = 85  |
    

Total emails = 90 + 10 + 15 + 85 = 200

From this, we calculate:

• Precision = TP / (TP + FP) = 90 / (90 + 15) = 0.857
• Recall = TP / (TP + FN) = 90 / (90 + 10) = 0.9
• F1 Score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.878

Precision vs Recall Tradeoff with Custom Transformers

Custom transformers can affect precision and recall differently. For example:

• If the transformer removes too many words, the model might miss spam emails (low recall).
• If it keeps too many irrelevant words, the model might wrongly mark good emails as spam (low precision).

Choosing the right transformer means balancing these. For spam detection, high precision is important to avoid losing good emails. For medical tests, high recall is key to catch all cases.

Good vs Bad Metric Values for Custom Transformers

Good:

• Precision and recall both above 0.8, showing balanced performance.
• F1 score close to or above 0.85, indicating good overall accuracy.
• Consistent transformation results on new data.

Bad:

• Precision or recall below 0.5, meaning many errors.
• F1 score below 0.6, showing poor balance.
• Transformer changes data unpredictably, causing model confusion.

Common Pitfalls in Metrics for Custom Transformers

• Accuracy paradox: High accuracy can be misleading if data is imbalanced. For example, if spam is rare, a model that always says "not spam" has high accuracy but is useless.
• Data leakage: If the transformer uses information from the test set during training, metrics will look better but won't work in real life.
• Overfitting: A transformer tuned too much on training data may not help on new data, causing metrics to drop.

Self-Check Question

Your model with a custom transformer has 98% accuracy but only 12% recall on fraud detection. Is it good for production?

Answer: No. The low recall means the model misses most fraud cases, which is dangerous. Even with high accuracy, the model fails its main job. You should improve recall before using it.

Key Result

Custom transformers should improve model precision, recall, and F1 score while ensuring consistent data transformation.