CatBoost in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - CatBoost

Which metric matters for CatBoost and WHY

CatBoost is a powerful tool for classification and regression. The metric you choose depends on your task:

• For classification: Use Accuracy to see overall correct predictions, but also Precision, Recall, and F1-score to understand how well it finds positive cases and avoids mistakes.
• For regression: Use Mean Squared Error (MSE) or Root Mean Squared Error (RMSE) to measure how close predictions are to actual values.

CatBoost handles categorical data well, so metrics that reflect real-world impact, like recall for rare events, are important.

Confusion Matrix Example for CatBoost Classification

    Actual \ Predicted | Positive | Negative
    -------------------|----------|---------
    Positive           |    80    |   20    
    Negative           |    10    |   90    
  

From this matrix:

• True Positives (TP) = 80
• False Positives (FP) = 10
• True Negatives (TN) = 90
• False Negatives (FN) = 20

Precision = 80 / (80 + 10) = 0.89

Recall = 80 / (80 + 20) = 0.80

F1-score = 2 * (0.89 * 0.80) / (0.89 + 0.80) ≈ 0.84

Precision vs Recall Tradeoff with CatBoost

Imagine CatBoost is used to detect spam emails:

• High Precision: Most emails marked as spam really are spam. Good to avoid losing important emails.
• High Recall: Most spam emails are caught. Good to keep inbox clean but may mark some good emails as spam.

Depending on what matters more, you can tune CatBoost to favor precision or recall.

What Good vs Bad Metrics Look Like for CatBoost

For a balanced classification task:

• Good: Accuracy > 85%, Precision and Recall both above 80%, F1-score above 0.8.
• Bad: Accuracy high but recall very low (missing many positives), or precision very low (many false alarms).

For regression:

• Good: Low MSE or RMSE close to zero.
• Bad: High error values showing poor predictions.

Common Metric Pitfalls with CatBoost

• Accuracy Paradox: High accuracy can be misleading if data is imbalanced.
• Data Leakage: If test data leaks into training, metrics look unrealistically good.
• Overfitting: Very high training accuracy but low test accuracy means model memorizes data, not learns patterns.
• Ignoring Class Imbalance: Not using precision/recall or F1 can hide poor performance on minority classes.

Self-Check: Is a Model with 98% Accuracy but 12% Recall on Fraud Good?

No, this model is not good for fraud detection.

Even though accuracy is high, recall is very low. This means it misses 88% of fraud cases, which is dangerous.

For fraud, catching as many frauds as possible (high recall) is critical, even if it means some false alarms.

Key Result

CatBoost evaluation depends on task; for classification, balance precision and recall to avoid missing positives or false alarms.