Sequential model API in TensorFlow - Model Metrics & Evaluation

Metrics & Evaluation - Sequential model API

Which metric matters for Sequential model API and WHY

The Sequential model API is often used for simple neural networks where layers are stacked one after another. The key metrics to evaluate such models depend on the task:

• For classification: Accuracy, Precision, Recall, and F1 score help understand how well the model predicts classes.
• For regression: Mean Squared Error (MSE) or Mean Absolute Error (MAE) show how close predictions are to actual values.

Choosing the right metric helps you know if your model is learning well and making useful predictions.

Confusion matrix example for classification

Imagine a Sequential model classifying emails as spam or not spam. Here is a confusion matrix from 100 emails:

      | Predicted Spam | Predicted Not Spam |
      |----------------|--------------------|
      | True Positives (TP) = 40           |
      | False Positives (FP) = 10          |
      | False Negatives (FN) = 5           |
      | True Negatives (TN) = 45           |
    

Totals: TP + FP + FN + TN = 40 + 10 + 5 + 45 = 100 emails

From this, we calculate:

• Precision = TP / (TP + FP) = 40 / (40 + 10) = 0.8
• Recall = TP / (TP + FN) = 40 / (40 + 5) = 0.8889
• Accuracy = (TP + TN) / Total = (40 + 45) / 100 = 0.85
• F1 Score = 2 * (Precision * Recall) / (Precision + Recall) ≈ 0.8421

Precision vs Recall tradeoff with examples

When using Sequential models for classification, you often balance precision and recall:

• High Precision: Means fewer false alarms. For example, a spam filter that rarely marks good emails as spam.
• High Recall: Means catching most positive cases. For example, a disease detector that finds almost all sick patients.

Depending on your goal, you might want to adjust the model or threshold to favor one metric over the other.

What "good" vs "bad" metric values look like for Sequential models

For a Sequential model used in classification:

• Good: Accuracy above 80%, Precision and Recall above 75%, balanced F1 score.
• Bad: Accuracy near random guess (e.g., 50% for two classes), very low Precision or Recall (below 50%), or large difference between Precision and Recall.

For regression tasks, good models have low MSE or MAE, meaning predictions are close to actual values.

Common pitfalls when evaluating Sequential models

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced (e.g., 95% accuracy by always predicting the majority class).
• Data leakage: When test data leaks into training, metrics look better but model fails in real use.
• Overfitting: Model performs very well on training data but poorly on new data, showing high training accuracy but low test accuracy.
• Ignoring metric choice: Using accuracy alone for imbalanced data can hide poor performance on minority classes.

Self-check question

Your Sequential model has 98% accuracy but only 12% recall on fraud detection. Is it good for production? Why or why not?

Answer: No, it is not good. Although accuracy is high, the model misses 88% of fraud cases (low recall). In fraud detection, catching fraud (high recall) is critical to avoid losses. This model would let most fraud go undetected.

Key Result

For Sequential models, balance precision and recall to ensure meaningful predictions beyond just accuracy.

Practice

1. What is the main purpose of the Sequential model API in TensorFlow?

easy

A. To visualize the training process of a model

B. To create complex models with multiple inputs and outputs

C. To perform data preprocessing before training

D. To build a model by stacking layers in a linear order

Solution

1. Step 1: Understand the Sequential API purpose

   The Sequential API is designed to build models by stacking layers one after another in a simple linear fashion.

2. Step 2: Compare options with the API's function

   Options B, C, and D describe other functionalities not related to the Sequential API's main purpose.

3. Final Answer:

   To build a model by stacking layers in a linear order -> Option D

4. Quick Check:

   Sequential API = linear stacking of layers [OK]

Hint: Sequential means layers stacked one after another [OK]

Common Mistakes:

• Confusing Sequential with Functional API for complex models
• Thinking Sequential handles data preprocessing
• Assuming Sequential is for visualization

2. Which of the following is the correct way to create a Sequential model with one dense layer of 10 units in TensorFlow?

easy

A. model = Sequential(Dense(10))

B. model = Sequential([Dense(10)])

C. model = Sequential().add(Dense(10))

D. model = Sequential.add(Dense(10))

Solution

1. Step 1: Recall correct Sequential model creation syntax

   The Sequential model can be created by passing a list of layers inside the constructor, e.g., Sequential([Dense(10)]).

2. Step 2: Check each option's syntax validity

   model = Sequential([Dense(10)]) uses the correct list syntax. model = Sequential(Dense(10)) misses the list brackets. model = Sequential().add(Dense(10)) is valid usage but requires assignment to a variable to keep the model reference. model = Sequential.add(Dense(10)) incorrectly calls add() on the class, not an instance.

3. Final Answer:

   model = Sequential([Dense(10)]) -> Option B

4. Quick Check:

   Sequential needs list of layers in constructor [OK]

Hint: Pass layers as a list inside Sequential() [OK]

Common Mistakes:

• Omitting brackets around layers list
• Calling add() on class instead of instance
• Chaining add() without assignment

3. What will be the output shape of the model after running this code?

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

model = Sequential([
    Dense(5, input_shape=(10,)),
    Dense(3)
])
print(model.output_shape)

medium

A. (None, 5)

B. (10, 3)

C. (None, 3)

D. (5, 3)

Solution

1. Step 1: Understand input and output shapes in Sequential

   The input shape is (10,), so the first Dense layer outputs (None, 5). The second Dense layer outputs (None, 3) because it has 3 units.

2. Step 2: Identify final output shape

   The model's output shape is the output of the last layer, which is (None, 3). None means batch size is flexible.

3. Final Answer:

   (None, 3) -> Option C

4. Quick Check:

   Last Dense units = output shape [OK]

Hint: Output shape matches last layer units with batch None [OK]

Common Mistakes:

• Confusing input shape with output shape
• Using batch size instead of None
• Mixing layer output shapes

4. Identify the error in this Sequential model code:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

model = Sequential()
model.add(Dense(10, input_shape=(5,)))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')
model.fit(x_train, y_train, epochs=5)

medium

A. x_train and y_train are not defined

B. Sequential model cannot use add() method

C. Loss function 'mse' is invalid

D. Missing import for optimizer

Solution

1. Step 1: Check code for missing definitions

   The code uses x_train and y_train in model.fit() but they are not defined anywhere, causing a runtime error.

2. Step 2: Verify other parts

   Optimizer 'adam' and loss 'mse' are valid strings. add() method is valid for Sequential instances. Imports are sufficient.

3. Final Answer:

   x_train and y_train are not defined -> Option A

4. Quick Check:

   Undefined training data causes error [OK]

Hint: Check if training data variables are defined before fit() [OK]

Common Mistakes:

• Assuming loss 'mse' is invalid
• Thinking add() method is not allowed
• Ignoring missing data variables

5. You want to build a Sequential model for a classification task with 3 classes. Which of the following is the best final layer and loss combination?

hard

A. Dense(3, activation='softmax') with loss='categorical_crossentropy'

B. Dense(1, activation='sigmoid') with loss='mean_squared_error'

C. Dense(3, activation='relu') with loss='binary_crossentropy'

D. Dense(3) with loss='sparse_categorical_crossentropy'

Solution

1. Step 1: Understand classification output requirements

   For 3 classes, the final layer should have 3 units with softmax activation to output class probabilities.

2. Step 2: Match appropriate loss function

   For one-hot encoded labels, 'categorical_crossentropy' is the correct loss function to use with softmax output.

3. Final Answer:

   Dense(3, activation='softmax') with loss='categorical_crossentropy' -> Option A

4. Quick Check:

   Softmax + categorical_crossentropy = multi-class classification [OK]

Hint: Use softmax + categorical_crossentropy for multi-class tasks [OK]

Common Mistakes:

• Using sigmoid for multi-class output
• Using mean squared error for classification
• Missing activation in final layer