Neural network architecture in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Neural network architecture

Which metric matters for Neural Network Architecture and WHY

When we build a neural network, we want to know how well it learns and predicts. The key metrics depend on the task:

• For classification: Accuracy, Precision, Recall, and F1 score tell us how well the network separates classes.
• For regression: Mean Squared Error (MSE) or Mean Absolute Error (MAE) show how close predictions are to real values.

These metrics help us decide if the network design (layers, neurons, activation) is good or needs change.

Confusion Matrix Example for Neural Network Classification

Imagine a neural network classifies emails as spam or not spam. Here is a confusion matrix:

      | Predicted Spam          | Predicted Not Spam     |
      |------------------------|-----------------------|
      | True Positive (TP) = 80 | False Negative (FN) = 20 |
      | False Positive (FP) = 10| True Negative (TN) = 90 |
    

Total samples = 80 + 20 + 10 + 90 = 200

From this, we calculate:

• Precision = 80 / (80 + 10) = 0.89
• Recall = 80 / (80 + 20) = 0.80
• Accuracy = (80 + 90) / 200 = 0.85
• F1 Score = 2 * (0.89 * 0.80) / (0.89 + 0.80) ≈ 0.84

Precision vs Recall Tradeoff in Neural Network Architecture

Changing the neural network's design or threshold affects precision and recall:

• High Precision: Few false alarms. Good when false positives are costly, like marking good emails as spam.
• High Recall: Few misses. Important when missing a positive case is bad, like detecting cancer.

Adjusting layers, neurons, or activation functions can improve one metric but may lower the other. We must balance based on the problem.

Good vs Bad Metric Values for Neural Network Architecture

For a well-designed neural network:

• Good: Accuracy above 85%, Precision and Recall above 80%, balanced F1 score.
• Bad: Accuracy near random guess (e.g., 50% for two classes), very low precision or recall (below 50%), or large difference between precision and recall.

Bad metrics suggest the architecture may be too simple, too complex, or not trained well.

Common Pitfalls in Neural Network Metrics

• Accuracy Paradox: High accuracy can be misleading if classes are imbalanced (e.g., 95% accuracy but always predicts the majority class).
• Data Leakage: Training data accidentally includes test data, inflating metrics falsely.
• Overfitting Indicators: Training accuracy very high but test accuracy low means the network memorizes instead of learning.
• Ignoring Recall or Precision: Focusing only on accuracy can hide poor detection of important classes.

Self-Check Question

Your neural network has 98% accuracy but only 12% recall on fraud cases. Is it good for production?

Answer: No. The low recall means it misses most fraud cases, which is dangerous. Despite high accuracy, the model fails to catch fraud well. You should improve recall before using it.

Key Result

Neural network metrics like precision, recall, and accuracy reveal how well the architecture learns and balances errors.

Practice

1. What does the architecture of a neural network mainly describe?

easy

A. The type of keyboard used to code the network

B. How layers and neurons are connected to learn from data

C. The speed of the computer running the network

D. The color of the network's interface

Solution

1. Step 1: Understand the role of architecture in neural networks

   The architecture defines the structure, including layers and neuron connections, which determines how the network learns.

2. Step 2: Eliminate unrelated options

   Options about color, speed, or keyboard do not relate to network design or learning.

3. Final Answer:

   How layers and neurons are connected to learn from data -> Option B

4. Quick Check:

   Neural network architecture = connections and layers [OK]

Hint: Think about what controls learning structure [OK]

Common Mistakes:

• Confusing hardware with architecture
• Thinking architecture is about UI design

2. Which of the following is the correct way to start a simple Sequential model in Python using Keras?

easy

A. model = Sequential()

B. model = sequence()

C. model = Sequential[]

D. model = Sequential{}

Solution

1. Step 1: Recall the correct syntax for creating a Sequential model

   In Keras, the Sequential model is created by calling Sequential() with parentheses.

2. Step 2: Identify incorrect syntax

   Using lowercase, square brackets, or curly braces is invalid for this class instantiation.

3. Final Answer:

   model = Sequential() -> Option A

4. Quick Check:

   Use parentheses to create Sequential model [OK]

Hint: Remember classes need parentheses to create instances [OK]

Common Mistakes:

• Using square or curly brackets
• Using lowercase 'sequence' instead of 'Sequential'

3. Given this code snippet, what will be the output shape of the model's last layer?

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

model = Sequential()
model.add(Dense(10, input_shape=(5,), activation='relu'))
model.add(Dense(3, activation='softmax'))
print(model.output_shape)

medium

A. (None, 3)

B. (5, 10)

C. (None, 10)

D. (3, 5)

Solution

1. Step 1: Understand the model layers and their output shapes

   The first Dense layer outputs shape (None, 10) because input is (5,) and units=10. The second Dense layer outputs (None, 3) because it has 3 units.

2. Step 2: Identify the final output shape

   The model's output shape is the output of the last layer, which is (None, 3). None means batch size is flexible.

3. Final Answer:

   (None, 3) -> Option A

4. Quick Check:

   Last Dense units = output shape [OK]

Hint: Output shape matches last layer units [OK]

Common Mistakes:

• Confusing input shape with output shape
• Ignoring batch size dimension

4. Identify the error in this neural network code snippet:

model = Sequential()
model.add(Dense(32, activation='relu'))
model.add(Dense(1))
model.compile(optimizer='adam', loss='categorical_crossentropy')

medium

A. Activation function 'relu' is invalid

B. Missing input_shape in first Dense layer

C. Loss function 'categorical_crossentropy' is incorrect for single output

D. Optimizer 'adam' is not supported

Solution

1. Step 1: Check the output layer and loss function compatibility

   The output layer has 1 unit, likely for regression or binary classification. 'categorical_crossentropy' expects multiple classes, so it's incorrect here.

2. Step 2: Verify other parts

   Input shape can be set later or inferred, 'relu' is valid, and 'adam' optimizer is supported.

3. Final Answer:

   Loss function 'categorical_crossentropy' is incorrect for single output -> Option C

4. Quick Check:

   Loss must match output type [OK]

Hint: Match loss to output layer type [OK]

Common Mistakes:

• Assuming input_shape is always mandatory
• Confusing loss functions for output types

5. You want to build a neural network for a multi-class classification problem with 4 classes. Which architecture is best suited?

hard

A. Input layer -> Dense(1, softmax) -> Dense(4, relu)

B. Input layer -> Dense(4, relu) -> Dense(1, sigmoid)

C. Input layer -> Dense(10, sigmoid) -> Dense(4, relu)

D. Input layer -> Dense(16, relu) -> Dense(4, softmax)

Solution

1. Step 1: Understand output layer requirements for multi-class classification

   For 4 classes, output layer should have 4 units with softmax activation to output class probabilities.

2. Step 2: Check each option's output layer and activations

   Input layer -> Dense(16, relu) -> Dense(4, softmax) ends with Dense(4, softmax), which is correct. Others have wrong units or activations for output.

3. Final Answer:

   Input layer -> Dense(16, relu) -> Dense(4, softmax) -> Option D

4. Quick Check:

   Output units = number of classes, softmax for multi-class [OK]

Hint: Output units = classes, use softmax for multi-class [OK]

Common Mistakes:

• Using sigmoid for multi-class output
• Wrong number of output units
• Misplacing activation functions