SavedModel format in TensorFlow - Model Metrics & Evaluation

Metrics & Evaluation - SavedModel format

Which metric matters for SavedModel format and WHY

The SavedModel format is a way to save and share trained TensorFlow models. It stores the model's architecture, weights, and computation graph so you can reuse it later or deploy it. The key metrics to check when using SavedModel are model accuracy and model loss before saving and after loading. This ensures the model saved is the same as the one loaded, with no loss in performance.

Why? Because the SavedModel format is about preserving the model's ability to make correct predictions. If accuracy or loss changes after saving and loading, it means the model was not saved or restored correctly.

Confusion matrix or equivalent visualization

Imagine you have a classification model saved in SavedModel format. After loading, you test it on 100 samples and get this confusion matrix:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP): 40 | False Negative (FN): 10 |
      | False Positive (FP): 5 | True Negative (TN): 45 |
    

Total samples = 40 + 10 + 5 + 45 = 100

From this, you can calculate:

• Precision = TP / (TP + FP) = 40 / (40 + 5) = 0.89
• Recall = TP / (TP + FN) = 40 / (40 + 10) = 0.80
• Accuracy = (TP + TN) / Total = (40 + 45) / 100 = 0.85

If these values match before saving and after loading, the SavedModel format worked well.

Precision vs Recall tradeoff with concrete examples

When you save and load a model using SavedModel, you want to keep the balance between precision and recall intact.

For example, if your model is a spam filter:

• High precision means few good emails are marked as spam (few false positives).
• High recall means most spam emails are caught (few false negatives).

If after loading the model, precision drops but recall stays high, the model might mark too many good emails as spam. This means the SavedModel format did not preserve the model perfectly.

So, checking precision and recall before and after saving helps ensure the model's behavior stays consistent.

What "good" vs "bad" metric values look like for SavedModel format

Good:

• Accuracy, precision, recall, and loss values before saving and after loading are nearly the same (differences less than 0.01).
• Confusion matrix counts match closely.
• Model predictions on test data are identical or very close.

Bad:

• Significant drop in accuracy or increase in loss after loading.
• Precision or recall changes drastically, indicating the model behaves differently.
• Confusion matrix counts differ greatly, showing prediction errors.
• Model fails to load or throws errors, meaning the SavedModel format is corrupted or incomplete.

Metrics pitfalls when using SavedModel format

• Data leakage: If test data leaks into training, metrics look better but model is not truly saved well.
• Overfitting: High training accuracy but low test accuracy can mislead you about model quality after saving.
• Version mismatch: Saving with one TensorFlow version and loading with another may cause errors or metric changes.
• Ignoring metric changes: Not checking metrics after loading can hide problems in the SavedModel.
• Confusion matrix miscalculation: Not summing TP, FP, TN, FN correctly leads to wrong precision/recall.

Self-check question

Your model saved in SavedModel format has 98% accuracy before saving but only 12% recall on fraud cases after loading. Is it good for production? Why or why not?

Answer: No, it is not good. Although accuracy is high, recall on fraud is very low. This means the model misses most fraud cases after loading, which is dangerous. The SavedModel format did not preserve the model's ability to detect fraud well. You should investigate saving/loading steps or model compatibility.

Key Result

SavedModel format must preserve model accuracy, precision, and recall before and after saving to ensure reliable predictions.

Practice

1. What does the TensorFlow SavedModel format primarily store?

easy

A. Only the model's training data

B. Only the model's weights

C. Only the model's optimizer state

D. The complete model architecture, weights, and training configuration

Solution

1. Step 1: Understand what SavedModel stores

   The SavedModel format saves the entire model, including architecture, weights, and training configuration.

2. Step 2: Compare options with SavedModel features

   Options A, B, and D only mention parts of the model or incorrect elements, but SavedModel stores all parts together as in C.

3. Final Answer:

   The complete model architecture, weights, and training configuration -> Option D

4. Quick Check:

   SavedModel = full model save [OK]

Hint: SavedModel saves everything needed to reuse the model [OK]

Common Mistakes:

• Thinking SavedModel saves only weights
• Confusing SavedModel with checkpoint files
• Assuming training data is saved inside SavedModel

2. Which of the following is the correct way to save a TensorFlow Keras model in SavedModel format?

easy

A. model.save('path/to/model', save_format='tf')

B. model.save('path/to/model', save_format='h5')

C. tf.save_model(model, 'path/to/model')

D. model.save_weights('path/to/model')

Solution

1. Step 1: Recall the save_format options

   TensorFlow Keras models can be saved in 'h5' or 'tf' formats. 'tf' means SavedModel format.

2. Step 2: Check each option's correctness

   model.save('path/to/model', save_format='h5') saves in HDF5 format, not SavedModel. model.save('path/to/model', save_format='tf') correctly uses save_format='tf'. tf.save_model(model, 'path/to/model') is invalid syntax. model.save_weights('path/to/model') saves only weights, not full model.

3. Final Answer:

   model.save('path/to/model', save_format='tf') -> Option A

4. Quick Check:

   save_format='tf' means SavedModel [OK]

Hint: Use save_format='tf' to save in SavedModel format [OK]

Common Mistakes:

• Using save_format='h5' instead of 'tf'
• Trying to use tf.save_model which doesn't exist
• Saving only weights instead of full model

3. What will be the output of the following code snippet?

import tensorflow as tf
model = tf.keras.Sequential([tf.keras.layers.Dense(1)])
model.save('my_model')
loaded_model = tf.keras.models.load_model('my_model')
print(type(loaded_model))

medium

A. <class 'tensorflow.python.keras.engine.sequential.Sequential'>

B. <class 'tensorflow.python.keras.engine.functional.Functional'>

C. AttributeError

D. None

Solution

1. Step 1: Understand model saving and loading

   The model is a Sequential model saved in SavedModel format and then loaded back.

2. Step 2: Check the type of loaded model

   Loading a Sequential model returns an instance of tf.keras.Sequential class, so the type is Sequential.

3. Final Answer:

   <class 'tensorflow.python.keras.engine.sequential.Sequential'> -> Option A

4. Quick Check:

   Loaded model type matches original Sequential [OK]

Hint: Loaded model keeps original class type (Sequential here) [OK]

Common Mistakes:

• Expecting Functional model type after loading Sequential
• Thinking load_model returns None
• Confusing error with AttributeError

4. You saved a model using model.save('model_path') but when loading with tf.keras.models.load_model('wrong_path'), you get an error. What is the likely cause?

medium

A. The model was not compiled before saving

B. The model was saved in HDF5 format, so load_model cannot find SavedModel files

C. The load path is incorrect or does not exist

D. The model weights were not saved

Solution

1. Step 1: Analyze the error cause

   Loading a model from a wrong or non-existing path causes a file not found error.

2. Step 2: Check other options

   The model was saved in HDF5 format, so load_model cannot find SavedModel files is unrelated to path errors. The model was not compiled before saving and D do not cause loading errors but affect training or evaluation.

3. Final Answer:

   The load path is incorrect or does not exist -> Option C

4. Quick Check:

   Wrong path causes load_model error [OK]

Hint: Check file path carefully when loading models [OK]

Common Mistakes:

• Assuming model must be compiled before saving
• Confusing missing weights with loading errors
• Ignoring file path correctness

5. You want to deploy a TensorFlow model to a server. Which of these steps correctly uses the SavedModel format to save and load the model for deployment?

hard

A. Save with tf.saved_model.save(model, 'deploy_model') and load with tf.saved_model.load('deploy_model')

B. Save with model.save('deploy_model', save_format='tf') and load with tf.keras.models.load_model('deploy_model')

C. Save with model.save('deploy_model.h5') and load with tf.keras.models.load_model('deploy_model.h5')

D. Save with model.save_weights('deploy_model') and load with model.load_weights('deploy_model')

Solution

1. Step 1: Identify correct save and load methods for deployment

   Using model.save with save_format='tf' saves in SavedModel format, suitable for deployment. Loading with tf.keras.models.load_model restores the full model.

2. Step 2: Evaluate other options

   Save with model.save_weights('deploy_model') and load with model.load_weights('deploy_model') saves only weights, not full model. Save with model.save('deploy_model.h5') and load with tf.keras.models.load_model('deploy_model.h5') saves in HDF5, which is less preferred for deployment. Save with tf.saved_model.save(model, 'deploy_model') and load with tf.saved_model.load('deploy_model') uses low-level API which returns a different object, not a Keras model.

3. Final Answer:

   Save with model.save('deploy_model', save_format='tf') and load with tf.keras.models.load_model('deploy_model') -> Option B

4. Quick Check:

   Use model.save with save_format='tf' for deployment [OK]

Hint: Use model.save with save_format='tf' and load_model for deployment [OK]

Common Mistakes:

• Saving only weights for deployment
• Using tf.saved_model.load which returns non-Keras object
• Saving in HDF5 format instead of SavedModel