Simple neural network with scikit-learn in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Simple neural network with scikit-learn

Which metric matters for this concept and WHY

When using a simple neural network with scikit-learn, the main goal is to see how well the model predicts new data. For classification tasks, accuracy is often used because it shows the percentage of correct predictions. However, accuracy alone can be misleading if classes are imbalanced. Therefore, precision, recall, and F1 score are important to understand the model's behavior on each class.

For regression tasks, metrics like mean squared error (MSE) or R-squared are used to measure how close predictions are to actual values.

Confusion matrix or equivalent visualization (ASCII)

For a binary classification example, the confusion matrix looks like this:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP)  | False Positive (FP) |
      | False Negative (FN) | True Negative (TN)  |
    

Example with numbers:

      | 50 | 10 |
      | 5  | 35 |
    

Here, total samples = 50 + 10 + 5 + 35 = 100.

Precision vs Recall tradeoff with concrete examples

Precision tells us how many of the predicted positives are actually positive. High precision means fewer false alarms.

Recall tells us how many of the actual positives were found. High recall means fewer misses.

Example: If the neural network is used for spam detection, high precision is important to avoid marking good emails as spam. If it is used for disease detection, high recall is important to catch as many sick patients as possible.

What "good" vs "bad" metric values look like for this use case

Good metrics for a simple neural network classification task might be:

• Accuracy above 85%
• Precision and recall both above 80%
• F1 score close to precision and recall

Bad metrics might be:

• Accuracy near 50% on balanced data (like random guessing)
• Precision very high but recall very low (model misses many positives)
• Recall very high but precision very low (model has many false alarms)

Metrics pitfalls

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced. For example, if 95% of data is negative, predicting all negative gives 95% accuracy but zero recall on positives.
• Data leakage: If test data leaks into training, metrics look unrealistically good.
• Overfitting indicators: Very high training accuracy but low test accuracy means the model memorizes training data but fails on new data.

Self-check question

Your simple neural network model has 98% accuracy but only 12% recall on the fraud class. Is it good for production? Why or why not?

Answer: No, it is not good. Although accuracy is high, the model misses 88% of fraud cases (low recall). For fraud detection, missing fraud is costly, so recall must be higher even if accuracy drops.

Key Result

For simple neural networks with scikit-learn, balance precision and recall to ensure meaningful performance beyond accuracy.