Model optimization (distillation, quantization) in NLP - Model Metrics & Evaluation

Metrics & Evaluation - Model optimization (distillation, quantization)

Which metric matters for Model optimization (distillation, quantization) and WHY

When optimizing models by distillation or quantization, the key metrics to watch are accuracy or task-specific performance (like F1 score for NLP tasks). This is because these methods reduce model size or speed up inference but can cause small drops in prediction quality. We want to keep the model as accurate as possible while making it smaller or faster.

Latency and model size are also important metrics here, but they are not about prediction quality. They measure how fast and small the model is after optimization.

Confusion matrix example after distillation

    Original model confusion matrix:
      TP=90  FP=10
      FN=5   TN=95

    Distilled model confusion matrix:
      TP=88  FP=12
      FN=7   TN=93

    Total samples = 90+10+5+95 = 200
    

Notice the small drop in true positives and increase in false negatives after distillation, showing a slight accuracy loss.

Precision vs Recall tradeoff in model optimization

When we optimize a model, sometimes precision or recall can drop. For example, quantization might make the model less sensitive, lowering recall (missing some true cases). Distillation might simplify the model, affecting precision (more false alarms).

Example: For a spam detector, if recall drops, some spam emails get through. If precision drops, good emails get marked as spam. We must balance these based on what matters more.

Good vs Bad metric values after optimization

Good: Accuracy or F1 score drops less than 1-2% compared to original, with significant size or speed gains.

Bad: Accuracy or F1 score drops more than 5%, causing poor predictions even if the model is smaller or faster.

Common pitfalls in metrics for model optimization

• Ignoring accuracy drop: Focusing only on size or speed and missing big accuracy loss.
• Not testing on real data: Optimized model might perform worse on real-world inputs.
• Overfitting to test set: Tuning optimization to a small test set can give misleading metrics.
• Confusing latency with accuracy: Faster model is good, but not if predictions become unreliable.

Self-check question

Your original NLP model has 95% accuracy. After quantization, accuracy drops to 92%, but inference speed doubles and model size halves. Is this good?

Answer: It depends on your use case. A 3% accuracy drop might be acceptable if speed and size improvements are critical, like on mobile devices. But if accuracy is crucial, this drop might be too large. Always balance metrics based on needs.

Key Result

Model optimization aims to keep accuracy or F1 score high while reducing size and latency, balancing tradeoffs carefully.

Practice

1. What is the main goal of model distillation in NLP?

easy

A. To increase the number of layers in a neural network

B. To add more training data for better accuracy

C. To convert text data into numerical vectors

D. To train a smaller model to mimic a larger model's behavior

Solution

1. Step 1: Understand model distillation concept

   Model distillation is about making a smaller model learn from a bigger, well-trained model.

2. Step 2: Identify the goal of distillation

   The goal is to keep performance while reducing model size and complexity.

3. Final Answer:

   To train a smaller model to mimic a larger model's behavior -> Option D

4. Quick Check:

   Distillation = smaller model copies bigger model [OK]

Hint: Distillation means small model learns from big model [OK]

Common Mistakes:

• Confusing distillation with adding layers
• Thinking distillation increases data size
• Mixing distillation with data preprocessing

2. Which of the following is the correct way to apply quantization to a model's weights in Python using PyTorch?

easy

A. model.quantize(weights=True)

B. torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)

C. torch.quantize(model, dtype=torch.float32)

D. torch.quantization(model, dtype=torch.int32)

Solution

1. Step 1: Recall PyTorch quantization syntax

   PyTorch uses torch.quantization.quantize_dynamic for dynamic quantization on layers like Linear.

2. Step 2: Check correct function and parameters

   torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8) correctly calls quantize_dynamic with model, target layers, and dtype torch.qint8.

3. Final Answer:

   torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8) -> Option B

4. Quick Check:

   PyTorch quantize_dynamic with Linear and qint8 = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8) [OK]

Hint: Use torch.quantization.quantize_dynamic for quantization [OK]

Common Mistakes:

• Using non-existent torch.quantize function
• Passing wrong dtype like float32 instead of qint8
• Calling quantization as a model method

3. Given the following code snippet for distillation, what will be the output loss value if the student model perfectly mimics the teacher model's outputs?

teacher_outputs = torch.tensor([0.1, 0.9])
student_outputs = torch.tensor([0.1, 0.9])
loss_fn = torch.nn.MSELoss()
loss = loss_fn(student_outputs, teacher_outputs)
print(loss.item())

medium

A. 0.0

B. 0.5

C. 1.0

D. Cannot compute due to shape mismatch

Solution

1. Step 1: Understand MSELoss calculation

   MSELoss calculates mean squared error between student and teacher outputs.

2. Step 2: Calculate loss for identical outputs

   Since student_outputs equals teacher_outputs, difference is zero, so loss is 0.0.

3. Final Answer:

   0.0 -> Option A

4. Quick Check:

   Identical outputs give zero MSE loss [OK]

Hint: Same outputs mean zero loss in MSE [OK]

Common Mistakes:

• Assuming loss is 1.0 by default
• Confusing loss with accuracy
• Thinking shape mismatch error occurs

4. You tried to quantize a model but got an error: AttributeError: 'MyModel' object has no attribute 'quantize'. What is the likely cause?

medium

A. The model class does not have a built-in quantize method

B. You forgot to import torch

C. Quantization only works on CPU, not GPU

D. The model is already quantized

Solution

1. Step 1: Analyze the error message

   The error says the model object lacks a 'quantize' method, meaning it is not defined.

2. Step 2: Understand quantization usage

   Quantization is applied via PyTorch functions, not as a model method, so calling model.quantize() causes error.

3. Final Answer:

   The model class does not have a built-in quantize method -> Option A

4. Quick Check:

   Quantize is a function, not a model method [OK]

Hint: Quantize via torch functions, not model methods [OK]

Common Mistakes:

• Trying to call quantize as model.quantize()
• Ignoring import errors
• Assuming quantization only works on CPU

5. You want to deploy a chatbot on a mobile device with limited memory and CPU. Which combination of model optimization techniques is best to reduce size and speed up inference without losing much accuracy?

hard

A. Use quantization first, then retrain the large model from scratch

B. Only increase the training data size to improve accuracy

C. Use distillation to train a smaller model, then apply quantization to reduce precision

D. Add more layers to the model and use float64 precision

Solution

1. Step 1: Identify constraints and goals

   Mobile devices need small, fast models with good accuracy.

2. Step 2: Choose suitable optimization techniques

   Distillation creates a smaller model; quantization reduces number precision to save space and speed up inference.

3. Step 3: Combine techniques for best effect

   Using distillation first then quantization is a common, effective approach.

4. Final Answer:

   Use distillation to train a smaller model, then apply quantization to reduce precision -> Option C

5. Quick Check:

   Distillation + quantization = small, fast, accurate model [OK]

Hint: Distill first, then quantize for mobile deployment [OK]

Common Mistakes:

• Ignoring quantization for mobile
• Adding layers increases size and slows down
• Retraining large model after quantization wastes effort