TensorFlow Lite conversion - Model Metrics & Evaluation

Metrics & Evaluation - TensorFlow Lite conversion

Which metric matters for TensorFlow Lite conversion and WHY

When converting a TensorFlow model to TensorFlow Lite, the key metrics to watch are model size, inference latency, and accuracy drop. Model size matters because smaller models run better on devices with limited storage. Latency is important because users want fast responses on mobile or embedded devices. Accuracy drop shows if the conversion changed the model's predictions negatively. We want to keep accuracy close to the original while improving size and speed.

Confusion matrix or equivalent visualization

Before and after conversion, compare the model's predictions on the same test data. For example, a confusion matrix for classification tasks can show if the TensorFlow Lite model makes more mistakes.

Original model confusion matrix:
TP=90  FP=10
FN=5   TN=95

TensorFlow Lite model confusion matrix:
TP=88  FP=12
FN=7   TN=93

Total samples = 90+10+5+95 = 200

This shows a slight drop in true positives and increase in false positives after conversion, indicating a small accuracy loss.

Precision vs Recall tradeoff with concrete examples

After conversion, precision and recall might change. For example, if the original model had precision 90/(90+10)=0.9 and recall 90/(90+5)=0.947, the TensorFlow Lite model might have precision 88/(88+12)=0.88 and recall 88/(88+7)=0.926.

This small drop means the converted model is slightly less precise and recalls fewer true positives. The tradeoff is acceptable if the model runs faster and smaller on the device.

What "good" vs "bad" metric values look like for TensorFlow Lite conversion

Good: Model size reduced by 50% or more, latency reduced by 30% or more, and accuracy drop less than 2%. Precision and recall remain above 85%.

Bad: Model size barely changes, latency stays high, or accuracy drops more than 5%, causing poor predictions. Precision or recall below 80% means the model is less reliable after conversion.

Metrics pitfalls

• Ignoring accuracy drop: Focusing only on size and speed can hide big accuracy losses.
• Data leakage: Testing on data used during training can give false high accuracy.
• Overfitting indicators: If the converted model performs well on training data but poorly on new data, it may be overfitting.
• Not testing on real device: Metrics measured on desktop may not reflect real device performance.

Self-check question

Your TensorFlow Lite model has 98% accuracy but 12% recall on fraud detection. Is it good for production? Why or why not?

Answer: No, it is not good. Although accuracy is high, recall is very low. This means the model misses most fraud cases, which is dangerous. For fraud detection, high recall is critical to catch as many frauds as possible.

Key Result

TensorFlow Lite conversion aims to reduce model size and latency with minimal accuracy loss, balancing precision and recall for reliable on-device performance.

Practice

1. What is the main purpose of converting a TensorFlow model to TensorFlow Lite format?

easy

A. To convert the model into a Python script

B. To increase the model's accuracy on large servers

C. To visualize the model architecture

D. To make the model smaller and faster for mobile and embedded devices

Solution

1. Step 1: Understand TensorFlow Lite's purpose

   TensorFlow Lite is designed to optimize models for mobile and embedded devices by reducing size and improving speed.

2. Step 2: Compare options

   Converting the model into a Python script, increasing accuracy on large servers, and visualizing the model architecture do not relate to model optimization for small devices. Only making the model smaller and faster for mobile and embedded devices correctly states the purpose.

3. Final Answer:

   To make the model smaller and faster for mobile and embedded devices -> Option D

4. Quick Check:

   TensorFlow Lite = smaller, faster models [OK]

Hint: Remember: TFLite = small, fast models for devices [OK]

Common Mistakes:

• Thinking TFLite improves accuracy on servers
• Confusing TFLite with model visualization tools
• Believing TFLite converts models to Python code

2. Which of the following is the correct way to create a TensorFlow Lite converter from a Keras model named model?

easy

A. converter = tf.lite.TFLiteConverter.convert(model)

B. converter = tf.lite.TFLiteConverter.from_keras_model(model)

C. converter = tf.lite.TFLiteConverter(model)

D. converter = tf.lite.TFLiteConverter.from_saved_model(model)

Solution

1. Step 1: Recall TensorFlow Lite converter syntax

   The correct method to create a converter from a Keras model is from_keras_model().

2. Step 2: Check each option

   converter = tf.lite.TFLiteConverter.from_keras_model(model) uses the correct method. converter = tf.lite.TFLiteConverter.convert(model) uses a non-existent method. converter = tf.lite.TFLiteConverter(model) tries to call the class directly without method. converter = tf.lite.TFLiteConverter.from_saved_model(model) is for SavedModel, not Keras model.

3. Final Answer:

   converter = tf.lite.TFLiteConverter.from_keras_model(model) -> Option B

4. Quick Check:

   Use from_keras_model() for Keras models [OK]

Hint: Use from_keras_model() to convert Keras models [OK]

Common Mistakes:

• Using convert() instead of from_keras_model()
• Calling TFLiteConverter class directly
• Using from_saved_model() for Keras models

3. Given the following code, what will be the type of the variable tflite_model after conversion?

import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()

medium

A. A bytes object containing the TensorFlow Lite model

B. A Python dictionary with model weights

C. A string with the model's JSON description

D. A TensorFlow Keras model object

Solution

1. Step 1: Understand the convert() method output

   The convert() method returns the TensorFlow Lite model as a bytes object, which can be saved to a .tflite file.

2. Step 2: Analyze options

   A TensorFlow Keras model object, a Python dictionary with model weights, and a string with the model's JSON description describe other data types not returned by convert(). A bytes object containing the TensorFlow Lite model correctly states the output type.

3. Final Answer:

   A bytes object containing the TensorFlow Lite model -> Option A

4. Quick Check:

   convert() returns bytes for .tflite file [OK]

Hint: convert() returns bytes, not model objects [OK]

Common Mistakes:

• Expecting a Keras model object after convert()
• Thinking convert() returns JSON or dict
• Confusing convert() output with model summary

4. You tried to convert a TensorFlow SavedModel using this code:

converter = tf.lite.TFLiteConverter.from_saved_model('my_model')
tflite_model = converter.convert()

But got an error: ValueError: SavedModel file does not exist at: my_model. What is the most likely cause?

medium

A. The path 'my_model' is incorrect or the folder does not exist

B. The model is not a Keras model

C. You forgot to import TensorFlow

D. The convert() method is not supported for SavedModel

Solution

1. Step 1: Understand the error message

   The error says the SavedModel file does not exist at the given path, indicating a path or folder issue.

2. Step 2: Evaluate options

   The path 'my_model' is incorrect or the folder does not exist matches the error cause. The model is not a Keras model is unrelated to file existence. You forgot to import TensorFlow would cause a different error. The convert() method is not supported for SavedModel is false; convert() works for SavedModel.

3. Final Answer:

   The path 'my_model' is incorrect or the folder does not exist -> Option A

4. Quick Check:

   Check model path if SavedModel not found [OK]

Hint: Check file path carefully when loading SavedModel [OK]

Common Mistakes:

• Assuming convert() doesn't work for SavedModel
• Ignoring file path correctness
• Confusing Keras model errors with SavedModel errors

5. You have a TensorFlow Keras model and want to convert it to TensorFlow Lite format with quantization to reduce size. Which sequence of steps is correct?

hard

A. Directly call convert() on the Keras model without creating a converter

B. Save model to disk, then use from_saved_model() without setting optimizations, then convert()

C. Create converter with from_keras_model(), set optimizations to DEFAULT, then convert()

D. Use from_concrete_function() with no optimizations, then convert()

Solution

1. Step 1: Identify correct converter creation for Keras model

   Use from_keras_model() to create the converter from a Keras model.

2. Step 2: Apply quantization by setting optimizations

   Set converter.optimizations = [tf.lite.Optimize.DEFAULT] to enable quantization.

3. Step 3: Convert the model

   Call convert() to get the quantized TensorFlow Lite model.

4. Final Answer:

   Create converter with from_keras_model(), set optimizations to DEFAULT, then convert() -> Option C

5. Quick Check:

   from_keras_model() + optimizations = quantized TFLite [OK]

Hint: Set converter.optimizations for quantization before convert() [OK]

Common Mistakes:

• Skipping setting optimizations for quantization
• Using from_saved_model() unnecessarily
• Calling convert() directly on model without converter