Saving pipelines (joblib, pickle) in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Saving pipelines (joblib, pickle)

Which metric matters for this concept and WHY

When saving machine learning pipelines using joblib or pickle, the key metric is model integrity. This means the saved pipeline should load back exactly as it was, preserving all steps and parameters so predictions remain the same. We check this by comparing predictions before saving and after loading. Accuracy or other performance metrics should not change. This ensures the pipeline is saved correctly and can be reused without errors.

Confusion matrix or equivalent visualization (ASCII)

Since saving pipelines is about preserving model behavior, we verify by comparing predictions before and after saving. For example, if the model predicts labels for 10 samples, the confusion matrix before saving and after loading should be identical.

    Before saving predictions: [1, 0, 1, 1, 0, 0, 1, 0, 1, 0]
    After loading predictions:  [1, 0, 1, 1, 0, 0, 1, 0, 1, 0]

    Confusion matrix (same for both):
    +-----+-----+
    | TP  | FP  |
    +-----+-----+
    | FN  | TN  |
    +-----+-----+

    TP = 5, FP = 0, FN = 0, TN = 5
    

Precision vs Recall tradeoff with concrete examples

Saving pipelines does not directly affect precision or recall. However, if the pipeline is corrupted during saving or loading, predictions may change, causing precision and recall to drop. For example, if a spam filter pipeline is saved incorrectly, it might mark good emails as spam (lower precision) or miss spam emails (lower recall). Thus, ensuring pipeline integrity preserves the original precision and recall.

What "good" vs "bad" metric values look like for this use case

Good: Predictions before saving and after loading are exactly the same. Accuracy, precision, recall, and F1 score remain unchanged. This means the pipeline was saved and loaded correctly.

Bad: Predictions differ after loading. Metrics drop significantly. This indicates the pipeline was corrupted or not saved properly, making it unreliable for future use.

Metrics pitfalls (accuracy paradox, data leakage, overfitting indicators)

• Corrupted save/load: Using incompatible versions of joblib or pickle can corrupt the pipeline.
• Data leakage: Saving pipelines that include data-dependent steps (like scaling on full data) without refitting on new data can cause misleading metrics.
• Overfitting: Saving a pipeline that overfits training data will preserve that behavior; metrics may look good on training but fail on new data.
• Accuracy paradox: High accuracy after loading does not guarantee pipeline integrity if the test set is unbalanced or small.

Self-check question

Your model pipeline was saved with joblib. After loading, the accuracy on the test set is 98%, but recall on the positive class dropped from 90% to 12%. Is the saved pipeline good for production? Why or why not?

Answer: No, the saved pipeline is not good. The large drop in recall means the model misses many positive cases after loading. This suggests the pipeline was corrupted or not saved properly. You must fix the saving/loading process to preserve model performance.

Key Result

Model integrity is key: predictions and metrics must remain unchanged before and after saving/loading pipelines.