Binary classification model in TensorFlow - Model Metrics & Evaluation

Metrics & Evaluation - Binary classification model

Which metric matters for this concept and WHY

In binary classification, the main goal is to correctly separate two groups, like "spam" vs "not spam" emails. The key metrics are Precision, Recall, and F1 score. Precision tells us how many predicted positives are actually correct. Recall tells us how many real positives we found. F1 score balances both. We also look at Accuracy to see overall correctness, but it can be misleading if classes are uneven.

Confusion matrix or equivalent visualization (ASCII)

      Confusion Matrix:

          Actual Positive   Actual Negative
    Predicted Positive    TP = 70          FP = 10
    Predicted Negative    FN = 20          TN = 100

    Total samples = TP + FP + FN + TN = 70 + 10 + 20 + 100 = 200
    

From this matrix, we calculate:

• Precision = TP / (TP + FP) = 70 / (70 + 10) = 0.875
• Recall = TP / (TP + FN) = 70 / (70 + 20) = 0.778
• F1 score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.823
• Accuracy = (TP + TN) / Total = (70 + 100) / 200 = 0.85

Precision vs Recall tradeoff with concrete examples

Imagine a spam filter. High precision means few good emails are wrongly marked as spam (low false alarms). High recall means catching most spam emails. If we focus only on recall, many good emails might be lost. So, for spam filters, precision is more important.

Now think about a cancer detector. Missing a cancer case (false negative) is very bad. So, high recall is critical to catch all sick patients, even if some healthy people get extra tests (lower precision). Here, recall matters more.

What "good" vs "bad" metric values look like for this use case

Good metrics for binary classification depend on the problem:

• Good: Precision and recall above 0.8, balanced F1 score, and accuracy above 0.8 usually mean the model works well.
• Bad: High accuracy but very low recall (e.g., 0.98 accuracy but 0.1 recall) means the model misses many positives. Or high recall but very low precision means many false alarms.

Always check precision and recall together, not just accuracy.

Metrics pitfalls

• Accuracy paradox: In imbalanced data, a model predicting only the majority class can have high accuracy but be useless.
• Data leakage: When test data leaks into training, metrics look unrealistically good.
• Overfitting indicators: Very high training accuracy but low test accuracy means the model memorizes training data and won't generalize.
• Ignoring class imbalance: Not adjusting metrics or thresholds when classes are uneven can mislead evaluation.

Self-check question

Your binary classification model has 98% accuracy but only 12% recall on the positive class (e.g., fraud). Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of positive cases, which is very risky for fraud detection. High accuracy is misleading here because most data is negative. You need to improve recall to catch more fraud cases.

Key Result

Precision, recall, and F1 score are key to evaluate binary classification models, especially with imbalanced data.