Mobile deployment (PyTorch Mobile) - Model Metrics & Evaluation

Metrics & Evaluation - Mobile deployment (PyTorch Mobile)

Which metric matters for Mobile deployment (PyTorch Mobile) and WHY

When deploying models on mobile devices, the key metrics are model size, inference latency, and accuracy.

Model size matters because mobile devices have limited storage. Smaller models fit better and load faster.

Inference latency is how fast the model makes predictions on the device. Faster means better user experience.

Accuracy shows how well the model predicts. We want to keep accuracy high even after making the model smaller or faster.

Balancing these metrics ensures the model works well and quickly on mobile phones.

Confusion matrix example for a mobile image classifier

    Actual \ Predicted | Cat | Dog | Bird
    -------------------|-----|-----|-----
    Cat                | 50  |  5  |  0
    Dog                |  3  | 45  |  2
    Bird               |  0  |  4  | 46

    Total samples = 155
    

From this matrix, we can calculate precision and recall for each class to check if the model still performs well on mobile.

Tradeoff: Accuracy vs Model Size and Speed on Mobile

Imagine you have a big, accurate model but it is slow and large. It drains battery and takes time to load.

If you shrink the model to make it faster and smaller, accuracy might drop.

For example, a model with 95% accuracy and 50MB size might be too big for mobile.

Reducing size to 10MB might lower accuracy to 90%, but the app runs smoothly.

Choosing the right balance depends on the app's needs: fast and small with slightly less accuracy, or bigger and slower but more accurate.

Good vs Bad metric values for Mobile deployment

Good:

• Model size < 20MB
• Inference latency < 100ms per prediction
• Accuracy > 90% (close to original model)

Bad:

• Model size > 100MB (too large for most phones)
• Inference latency > 500ms (slow response)
• Accuracy < 80% (too many wrong predictions)

Common pitfalls in Mobile deployment metrics

• Ignoring latency: A model with good accuracy but slow speed frustrates users.
• Overfitting: Model performs well on test data but poorly on real mobile data.
• Data leakage: Training data too similar to test data inflates accuracy falsely.
• Not testing on device: Metrics from desktop may not reflect mobile performance.
• Ignoring battery impact: Heavy models drain battery quickly.

Self-check question

Your mobile model has 98% accuracy but takes 800ms per prediction and is 120MB in size. Is it good for mobile deployment? Why or why not?

Answer: No, it is not good. Although accuracy is high, the model is too large and slow for mobile devices. It will cause delays and use too much storage, harming user experience.

Key Result

For mobile deployment, balancing model size, inference speed, and accuracy is key to good user experience.

Practice

1. What is the main advantage of using PyTorch Mobile for AI models on phones?

easy

A. It converts models to TensorFlow format

B. It requires an internet connection to work

C. It runs models directly on the phone for faster response and privacy

D. It only supports training models on the phone

Solution

1. Step 1: Understand PyTorch Mobile purpose

   PyTorch Mobile allows running AI models directly on mobile devices without needing a server.

2. Step 2: Identify benefits of local model execution

   Running models locally improves speed and keeps data private since no internet is needed.

3. Final Answer:

   It runs models directly on the phone for faster response and privacy -> Option C

4. Quick Check:

   Local model execution = faster and private [OK]

Hint: Remember: Mobile deployment means running models on the device [OK]

Common Mistakes:

• Thinking PyTorch Mobile needs internet
• Confusing training with inference on mobile
• Mixing PyTorch Mobile with TensorFlow

2. Which PyTorch function is used to convert a model to TorchScript for mobile deployment?

easy

A. torch.jit.trace()

B. torch.save()

C. torch.load()

D. torch.nn.Module()

Solution

1. Step 1: Recall TorchScript conversion methods

   PyTorch uses torch.jit.trace() or torch.jit.script() to convert models to TorchScript.

2. Step 2: Identify correct function for conversion

   torch.jit.trace() records operations to create a TorchScript model for mobile use.

3. Final Answer:

   torch.jit.trace() -> Option A

4. Quick Check:

   Model conversion = torch.jit.trace() [OK]

Hint: Trace or script to convert model for mobile [OK]

Common Mistakes:

• Using torch.save() which only saves models
• Confusing torch.load() which loads models
• Using torch.nn.Module() which defines models

3. What will be the output type of this code snippet?

import torch
class MyModel(torch.nn.Module):
    def forward(self, x):
        return x * 2
model = MyModel()
traced_model = torch.jit.trace(model, torch.tensor([1]))
print(type(traced_model))

medium

A. <class 'torch.nn.Module'>

B. <class 'torch.jit.ScriptModule'>

C. <class 'torch.jit.TopLevelTracedModule'>

D. <class 'torch.jit.TracedModule'>

Solution

1. Step 1: Understand torch.jit.trace output type

   Tracing a model returns a TorchScript object of type ScriptModule in PyTorch.

2. Step 2: Check printed type

   Printing type(traced_model) shows <class 'torch.jit.ScriptModule'> indicating the traced script module.

3. Final Answer:

   <class 'torch.jit.ScriptModule'> -> Option B

4. Quick Check:

   Tracing returns ScriptModule [OK]

Hint: Trace returns ScriptModule type [OK]

Common Mistakes:

• Assuming output is TopLevelTracedModule
• Confusing traced model with normal nn.Module
• Choosing non-existent TracedModule class

4. Identify the error in this PyTorch Mobile model saving code:

import torch
model = torch.nn.Linear(2, 2)
traced = torch.jit.trace(model, torch.randn(2))
torch.save(traced, 'model.pt')
loaded = torch.jit.load('model.pt')
print(loaded(torch.randn(2)))

medium

A. torch.save() cannot save traced models

B. torch.jit.load() cannot load saved models

C. Input tensor shape for tracing is incorrect

D. Model must be scripted, not traced

Solution

1. Step 1: Identify the saving issue

   TorchScript models from torch.jit.trace() must use torch.jit.save(traced, 'model.pt'); torch.save() fails as ScriptModules are not picklable.

2. Step 2: Verify other components

   torch.randn(2) matches Linear(2,2) input shape; torch.jit.load() works after proper save.

3. Final Answer:

   torch.save() cannot save traced models -> Option A

4. Quick Check:

   TorchScript requires torch.jit.save() [OK]

Hint: torch.jit.save() for traced/ScriptModules [OK]

Common Mistakes:

• torch.save() doesn't pickle ScriptModules
• torch.jit.load() works correctly
• Traced models save with torch.jit.save()

5. You want to deploy a PyTorch model on Android. Which sequence correctly prepares the model for mobile use?

hard

A. Train model -> Convert to TorchScript with torch.save() -> Save as .pt -> Load in Android app

B. Train model -> Save as .pt -> Load directly in Android app -> Convert to TorchScript

C. Train model -> Convert to ONNX -> Save as .pt -> Load in Android app

D. Train model -> Convert to TorchScript with torch.jit.trace() -> Save as .pt -> Load in Android app

Solution

1. Step 1: Understand model conversion for mobile

   PyTorch Mobile requires models in TorchScript format, created by tracing or scripting.

2. Step 2: Identify correct conversion and saving steps

   Tracing with torch.jit.trace(), saving as .pt, then loading in Android app is the correct flow.

3. Final Answer:

   Train model -> Convert to TorchScript with torch.jit.trace() -> Save as .pt -> Load in Android app -> Option D

4. Quick Check:

   Trace -> save -> load for mobile deployment [OK]

Hint: Trace or script, save .pt, then load on mobile [OK]

Common Mistakes:

• Trying to load raw PyTorch model on mobile
• Using ONNX instead of TorchScript for PyTorch Mobile
• Saving before converting to TorchScript