Simple neural network with scikit-learn in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Simple neural network with scikit-learn

Which metric matters for this concept and WHY

When using a simple neural network with scikit-learn, the main goal is to see how well the model predicts new data. For classification tasks, accuracy is often used because it shows the percentage of correct predictions. However, accuracy alone can be misleading if classes are imbalanced. Therefore, precision, recall, and F1 score are important to understand the model's behavior on each class.

For regression tasks, metrics like mean squared error (MSE) or R-squared are used to measure how close predictions are to actual values.

Confusion matrix or equivalent visualization (ASCII)

For a binary classification example, the confusion matrix looks like this:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP)  | False Positive (FP) |
      | False Negative (FN) | True Negative (TN)  |
    

Example with numbers:

      | 50 | 10 |
      | 5  | 35 |
    

Here, total samples = 50 + 10 + 5 + 35 = 100.

Precision vs Recall tradeoff with concrete examples

Precision tells us how many of the predicted positives are actually positive. High precision means fewer false alarms.

Recall tells us how many of the actual positives were found. High recall means fewer misses.

Example: If the neural network is used for spam detection, high precision is important to avoid marking good emails as spam. If it is used for disease detection, high recall is important to catch as many sick patients as possible.

What "good" vs "bad" metric values look like for this use case

Good metrics for a simple neural network classification task might be:

• Accuracy above 85%
• Precision and recall both above 80%
• F1 score close to precision and recall

Bad metrics might be:

• Accuracy near 50% on balanced data (like random guessing)
• Precision very high but recall very low (model misses many positives)
• Recall very high but precision very low (model has many false alarms)

Metrics pitfalls

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced. For example, if 95% of data is negative, predicting all negative gives 95% accuracy but zero recall on positives.
• Data leakage: If test data leaks into training, metrics look unrealistically good.
• Overfitting indicators: Very high training accuracy but low test accuracy means the model memorizes training data but fails on new data.

Self-check question

Your simple neural network model has 98% accuracy but only 12% recall on the fraud class. Is it good for production? Why or why not?

Answer: No, it is not good. Although accuracy is high, the model misses 88% of fraud cases (low recall). For fraud detection, missing fraud is costly, so recall must be higher even if accuracy drops.

Key Result

For simple neural networks with scikit-learn, balance precision and recall to ensure meaningful performance beyond accuracy.

Practice

1. What is the main purpose of using MLPClassifier in scikit-learn?

easy

A. To perform data scaling and normalization

B. To create and train a simple neural network for classification tasks

C. To visualize data distributions

D. To split data into training and testing sets

Solution

1. Step 1: Understand the role of MLPClassifier

   MLPClassifier is a tool in scikit-learn designed to build and train neural networks for classification problems.

2. Step 2: Compare options with MLPClassifier's purpose

   The other options describe tasks like scaling, visualization, and data splitting, which are not the main function of MLPClassifier.

3. Final Answer:

   To create and train a simple neural network for classification tasks -> Option B

4. Quick Check:

   MLPClassifier = neural network classifier [OK]

Hint: MLPClassifier builds neural nets for classification [OK]

Common Mistakes:

• Confusing MLPClassifier with data preprocessing tools
• Thinking it is for regression only
• Assuming it splits data automatically

2. Which of the following is the correct way to import the MLPClassifier from scikit-learn?

easy

A. import neural_network.MLPClassifier from sklearn

B. import MLPClassifier from sklearn.network

C. from sklearn.ml import MLPClassifier

D. from sklearn.neural_network import MLPClassifier

Solution

1. Step 1: Recall the correct module path for MLPClassifier

   MLPClassifier is located in the sklearn.neural_network module.

2. Step 2: Check each import statement for syntax and correctness

   The other options have incorrect module names or import syntax.

3. Final Answer:

   from sklearn.neural_network import MLPClassifier -> Option D

4. Quick Check:

   Correct import = from sklearn.neural_network import MLPClassifier [OK]

Hint: MLPClassifier is in sklearn.neural_network module [OK]

Common Mistakes:

• Using wrong module names like sklearn.ml
• Incorrect import syntax
• Confusing neural_network with network

3. What will be the output of the following code snippet?

from sklearn.neural_network import MLPClassifier
X = [[0., 0.], [1., 1.]]
y = [0, 1]
clf = MLPClassifier(hidden_layer_sizes=(2,), max_iter=1000, random_state=1)
clf.fit(X, y)
pred = clf.predict([[2., 2.]])
print(pred)

medium

A. [0]

B. Error due to wrong input shape

C. [1]

D. [2]

Solution

1. Step 1: Understand the training data and labels

   The model is trained on two points: [0., 0.] labeled 0 and [1., 1.] labeled 1.

2. Step 2: Predict the label for input [2., 2.]

   Since [2., 2.] is closer to [1., 1.] which has label 1, the model predicts 1.

3. Final Answer:

   [1] -> Option C

4. Quick Check:

   Input near class 1 predicts 1 [OK]

Hint: Prediction matches closest trained label [OK]

Common Mistakes:

• Assuming output is 0 by default
• Confusing input shape errors
• Ignoring random_state effect

4. Identify the error in the following code snippet:

from sklearn.neural_network import MLPClassifier
X = [[0, 0], [1, 1]]
y = [0, 1]
clf = MLPClassifier(hidden_layer_sizes=2, max_iter=500)
clf.fit(X, y)

medium

A. hidden_layer_sizes should be a tuple, not an integer

B. max_iter value is too low

C. X and y have mismatched lengths

D. MLPClassifier requires data scaling before fitting

Solution

1. Step 1: Check the hidden_layer_sizes parameter

   hidden_layer_sizes expects a tuple indicating the number of neurons per hidden layer, e.g., (2,), not an integer.

2. Step 2: Verify other parameters and data

   max_iter=500 is valid, X and y lengths match, and scaling is recommended but not mandatory.

3. Final Answer:

   hidden_layer_sizes should be a tuple, not an integer -> Option A

4. Quick Check:

   hidden_layer_sizes = tuple [OK]

Hint: hidden_layer_sizes must be tuple like (2,) [OK]

Common Mistakes:

• Passing integer instead of tuple for hidden_layer_sizes
• Assuming max_iter causes error
• Thinking scaling is mandatory for fit

5. You want to improve your MLPClassifier model's accuracy on a complex dataset. Which of these changes is most likely to help?

hard

A. Increase the number of neurons in hidden layers and train longer

B. Reduce the number of hidden layers to zero

C. Remove the activation function to speed up training

D. Use fewer training samples to avoid overfitting

Solution

1. Step 1: Understand model complexity and training

   Increasing neurons and training longer allows the model to learn more complex patterns.

2. Step 2: Evaluate other options

   Reducing hidden layers to zero removes learning capacity, removing activation breaks non-linearity, and fewer samples usually reduce accuracy.

3. Final Answer:

   Increase the number of neurons in hidden layers and train longer -> Option A

4. Quick Check:

   More neurons + training = better learning [OK]

Hint: More neurons and training improve complex learning [OK]

Common Mistakes:

• Thinking fewer layers always improve accuracy
• Removing activation functions to simplify
• Using less data to prevent overfitting blindly