Model optimization (quantization, pruning) in PyTorch - Model Metrics & Evaluation

Metrics & Evaluation - Model optimization (quantization, pruning)

Which metric matters for Model optimization (quantization, pruning) and WHY

When optimizing models by quantization or pruning, the key metrics to watch are model accuracy and inference latency. Accuracy tells us if the model still makes good predictions after optimization. Latency shows how fast the model runs, which is often the goal of optimization. We also check model size to see how much memory is saved. Balancing these metrics helps us keep the model useful while making it smaller and faster.

Confusion matrix example after pruning

    Original model confusion matrix:
      TP=90  FP=10
      FN=5   TN=95

    After pruning:
      TP=85  FP=15
      FN=10  TN=90

    Total samples = 200

    Precision before pruning = 90 / (90 + 10) = 0.9
    Recall before pruning = 90 / (90 + 5) = 0.947

    Precision after pruning = 85 / (85 + 15) = 0.85
    Recall after pruning = 85 / (85 + 10) = 0.895
    

This shows a slight drop in precision and recall after pruning, which is common but should be minimal.

Tradeoff: Accuracy vs Model Size and Speed

Quantization and pruning reduce model size and speed up inference but can lower accuracy. For example:

• Quantization: Converts weights from 32-bit floats to 8-bit integers. This shrinks model size and speeds up calculations but may cause small accuracy loss.
• Pruning: Removes less important connections. This reduces size and computation but can remove useful information, lowering accuracy.

We must decide how much accuracy loss is acceptable for the gain in speed and size. For mobile apps, smaller and faster models are often worth a small accuracy drop.

Good vs Bad metric values for Model optimization

Good:

• Accuracy drop < 1-2% after optimization
• Model size reduced by 50% or more
• Inference latency reduced by 30% or more

Bad:

• Accuracy drops more than 5%
• Minimal size or speed improvement
• Model becomes unstable or unpredictable

Common pitfalls in Model optimization metrics

• Ignoring accuracy drop: Focusing only on size/speed can break the model.
• Data leakage: Testing on data seen during training can hide accuracy loss.
• Overfitting to optimization: Tweaking too much on test data can give false confidence.
• Not measuring latency on target device: Speed gains on desktop may not appear on mobile.

Self-check question

Your model after pruning has 98% accuracy but recall on the positive class dropped to 12%. Is it good for production? Why or why not?

Answer: No, it is not good. Even though overall accuracy is high, the very low recall means the model misses most positive cases. For example, if detecting fraud, missing 88% of fraud cases is dangerous. High recall is critical in such tasks.

Key Result

Model optimization aims to reduce size and latency while keeping accuracy loss minimal, typically under 2%.

Practice

1. What is the main goal of model pruning in PyTorch?

easy

A. Add more training data to improve accuracy

B. Increase the number of layers in the model

C. Change the model architecture to a different type

D. Remove less important weights to make the model smaller

Solution

1. Step 1: Understand pruning concept

   Pruning means removing weights that do not contribute much to the model's output.

2. Step 2: Identify pruning effect

   Removing these weights makes the model smaller and faster without changing its structure.

3. Final Answer:

   Remove less important weights to make the model smaller -> Option D

4. Quick Check:

   Pruning = Remove less important weights [OK]

Hint: Pruning cuts unimportant weights to shrink model size [OK]

Common Mistakes:

• Thinking pruning adds layers
• Confusing pruning with data augmentation
• Believing pruning changes model type

2. Which PyTorch function is used to apply pruning to a layer?

easy

A. torch.nn.utils.prune.l1_unstructured

B. torch.quantization.quantize_dynamic

C. torch.optim.Adam

D. torch.nn.functional.relu

Solution

1. Step 1: Identify pruning functions in PyTorch

   PyTorch pruning utilities are in torch.nn.utils.prune module.

2. Step 2: Match pruning function

   l1_unstructured is a common pruning method that removes weights based on L1 norm.

3. Final Answer:

   torch.nn.utils.prune.l1_unstructured -> Option A

4. Quick Check:

   Pruning function = l1_unstructured [OK]

Hint: Pruning uses torch.nn.utils.prune functions [OK]

Common Mistakes:

• Confusing pruning with quantization functions
• Choosing optimizer functions by mistake
• Selecting activation functions instead of pruning

3. What will be the output shape of the model's weight tensor after applying this pruning code?

import torch
import torch.nn.utils.prune as prune

layer = torch.nn.Linear(4, 3)
prune.l1_unstructured(layer, name='weight', amount=0.5)
print(layer.weight.shape)

medium

A. torch.Size([4, 3]) with half weights removed

B. torch.Size([3, 4]) with half weights removed physically

C. torch.Size([3, 4]) with some weights zeroed

D. torch.Size([4, 3]) unchanged

Solution

1. Step 1: Check layer weight shape

   A Linear layer with input 4 and output 3 has weight shape (3, 4).

2. Step 2: Understand pruning effect on weight tensor

   Pruning sets some weights to zero but does not change tensor shape.

3. Final Answer:

   torch.Size([3, 4]) with some weights zeroed -> Option C

4. Quick Check:

   Pruning zeros weights, shape stays same [OK]

Hint: Pruning zeros weights but keeps tensor shape [OK]

Common Mistakes:

• Thinking pruning changes tensor shape
• Confusing input/output dimensions order
• Assuming weights are physically removed

4. Identify the error in this PyTorch pruning code:

import torch.nn.utils.prune as prune
layer = torch.nn.Linear(5, 2)
prune.l1_unstructured(layer, name='bias', amount=0.3)

medium

A. Missing import for torch

B. Layer input and output sizes are invalid

C. Amount must be an integer, not float

D. Cannot prune bias with l1_unstructured

Solution

1. Step 1: Examine the code imports

   The code imports only prune but attempts to use torch.nn.Linear without importing torch.

2. Step 2: Identify the runtime error

   This causes NameError: name 'torch' is not defined when creating the layer.

3. Final Answer:

   Missing import for torch -> Option A

4. Quick Check:

   Missing torch import causes NameError [OK]

Hint: Import torch before using torch.nn [OK]

Common Mistakes:

• Overlooking missing torch import
• Thinking pruning bias is unsupported
• Assuming amount must be integer

5. You want to speed up inference on a PyTorch model by reducing number precision. Which approach should you use?

hard

A. Increase batch size during training

B. Apply dynamic quantization to convert weights to lower precision

C. Prune 90% of the model weights randomly

D. Add dropout layers to the model

Solution

1. Step 1: Understand quantization purpose

   Quantization reduces number precision (e.g., float32 to int8) to speed up inference.

2. Step 2: Identify correct method

   Dynamic quantization in PyTorch converts weights to lower precision without retraining.

3. Step 3: Compare other options

   Pruning removes weights but may not speed inference as much; batch size and dropout affect training, not inference speed.

4. Final Answer:

   Apply dynamic quantization to convert weights to lower precision -> Option B

5. Quick Check:

   Quantization = lower precision for speed [OK]

Hint: Quantization reduces precision to speed inference [OK]

Common Mistakes:

• Confusing pruning with quantization
• Thinking batch size affects model speed directly
• Adding dropout to speed inference