Model optimization (distillation, quantization) in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Model Distillation Purpose

What is the main goal of model distillation in NLP?

A To train a smaller model to mimic a larger model's behavior

B To increase the number of parameters in the model for better accuracy

C To add noise to the training data to improve robustness

D To convert a model into a rule-based system

💡 Hint

Think about how a big model can help a smaller one learn.

Model distillation trains a smaller model (student) to imitate a larger, well-trained model (teacher), keeping performance but reducing size.

❓ Predict Output

intermediate

Output of Quantization Code Snippet

What is the output shape of the quantized model's embedding layer weights after applying 8-bit quantization?

NLP

import torch
import torch.nn as nn

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding = nn.Embedding(1000, 64)

model = SimpleModel()
quantized_model = torch.quantization.quantize_dynamic(
    model, {nn.Embedding}, dtype=torch.qint8
)
weight_shape = quantized_model.embedding.weight.shape
print(weight_shape)

A AttributeError

B (64, 1000)

C (1000, 32)

D (1000, 64)

💡 Hint

Quantization changes data type but not tensor shape.

The quantized embedding weights keep the original shape (1000, 64). Quantization changes precision, not dimensions.

❓ Model Choice

advanced

Choosing a Model for Distillation

You want to distill a large BERT model into a smaller one for mobile deployment. Which student model architecture is best suited?

A A recurrent neural network with LSTM cells

B A convolutional neural network designed for images

C A smaller BERT model with fewer layers and parameters

D A large transformer model with more layers than the teacher

💡 Hint

Student model should be similar but smaller than the teacher.

Distillation works best when the student model architecture is similar but smaller, like a smaller BERT variant, to capture the teacher's knowledge efficiently.

❓ Hyperparameter

advanced

Key Hyperparameter in Quantization

Which hyperparameter is critical to set correctly when applying post-training quantization to an NLP model?

A The learning rate of the optimizer

B The number of bits used to represent weights and activations

C The dropout rate in the model layers

D The batch size during training

💡 Hint

Quantization precision depends on this setting.

The bit-width (e.g., 8-bit) determines how finely weights and activations are represented, affecting model size and accuracy.

❓ Metrics

expert

Evaluating Distilled Model Performance

After distilling a large NLP model, which metric best shows if the smaller model retained the teacher's knowledge effectively?

A Accuracy on a held-out validation set compared to the teacher model

B Training loss of the student model on the training data

C Number of parameters in the student model

D Inference time on a GPU

💡 Hint

Think about measuring how well the student predicts compared to the teacher.

Accuracy on validation data shows if the student model learned to mimic the teacher's predictions well, indicating successful distillation.

Practice

1. What is the main goal of model distillation in NLP?

easy

A. To increase the number of layers in a neural network

B. To add more training data for better accuracy

C. To convert text data into numerical vectors

D. To train a smaller model to mimic a larger model's behavior

Solution

1. Step 1: Understand model distillation concept

   Model distillation is about making a smaller model learn from a bigger, well-trained model.

2. Step 2: Identify the goal of distillation

   The goal is to keep performance while reducing model size and complexity.

3. Final Answer:

   To train a smaller model to mimic a larger model's behavior -> Option D

4. Quick Check:

   Distillation = smaller model copies bigger model [OK]

Hint: Distillation means small model learns from big model [OK]

Common Mistakes:

• Confusing distillation with adding layers
• Thinking distillation increases data size
• Mixing distillation with data preprocessing

2. Which of the following is the correct way to apply quantization to a model's weights in Python using PyTorch?

easy

A. model.quantize(weights=True)

B. torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)

C. torch.quantize(model, dtype=torch.float32)

D. torch.quantization(model, dtype=torch.int32)

Solution

1. Step 1: Recall PyTorch quantization syntax

   PyTorch uses torch.quantization.quantize_dynamic for dynamic quantization on layers like Linear.

2. Step 2: Check correct function and parameters

   torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8) correctly calls quantize_dynamic with model, target layers, and dtype torch.qint8.

3. Final Answer:

   torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8) -> Option B

4. Quick Check:

   PyTorch quantize_dynamic with Linear and qint8 = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8) [OK]

Hint: Use torch.quantization.quantize_dynamic for quantization [OK]

Common Mistakes:

• Using non-existent torch.quantize function
• Passing wrong dtype like float32 instead of qint8
• Calling quantization as a model method

3. Given the following code snippet for distillation, what will be the output loss value if the student model perfectly mimics the teacher model's outputs?

teacher_outputs = torch.tensor([0.1, 0.9])
student_outputs = torch.tensor([0.1, 0.9])
loss_fn = torch.nn.MSELoss()
loss = loss_fn(student_outputs, teacher_outputs)
print(loss.item())

medium

A. 0.0

B. 0.5

C. 1.0

D. Cannot compute due to shape mismatch

Solution

1. Step 1: Understand MSELoss calculation

   MSELoss calculates mean squared error between student and teacher outputs.

2. Step 2: Calculate loss for identical outputs

   Since student_outputs equals teacher_outputs, difference is zero, so loss is 0.0.

3. Final Answer:

   0.0 -> Option A

4. Quick Check:

   Identical outputs give zero MSE loss [OK]

Hint: Same outputs mean zero loss in MSE [OK]

Common Mistakes:

• Assuming loss is 1.0 by default
• Confusing loss with accuracy
• Thinking shape mismatch error occurs

4. You tried to quantize a model but got an error: AttributeError: 'MyModel' object has no attribute 'quantize'. What is the likely cause?

medium

A. The model class does not have a built-in quantize method

B. You forgot to import torch

C. Quantization only works on CPU, not GPU

D. The model is already quantized

Solution

1. Step 1: Analyze the error message

   The error says the model object lacks a 'quantize' method, meaning it is not defined.

2. Step 2: Understand quantization usage

   Quantization is applied via PyTorch functions, not as a model method, so calling model.quantize() causes error.

3. Final Answer:

   The model class does not have a built-in quantize method -> Option A

4. Quick Check:

   Quantize is a function, not a model method [OK]

Hint: Quantize via torch functions, not model methods [OK]

Common Mistakes:

• Trying to call quantize as model.quantize()
• Ignoring import errors
• Assuming quantization only works on CPU

5. You want to deploy a chatbot on a mobile device with limited memory and CPU. Which combination of model optimization techniques is best to reduce size and speed up inference without losing much accuracy?

hard

A. Use quantization first, then retrain the large model from scratch

B. Only increase the training data size to improve accuracy

C. Use distillation to train a smaller model, then apply quantization to reduce precision

D. Add more layers to the model and use float64 precision

Solution

1. Step 1: Identify constraints and goals

   Mobile devices need small, fast models with good accuracy.

2. Step 2: Choose suitable optimization techniques

   Distillation creates a smaller model; quantization reduces number precision to save space and speed up inference.

3. Step 3: Combine techniques for best effect

   Using distillation first then quantization is a common, effective approach.

4. Final Answer:

   Use distillation to train a smaller model, then apply quantization to reduce precision -> Option C

5. Quick Check:

   Distillation + quantization = small, fast, accurate model [OK]

Hint: Distill first, then quantize for mobile deployment [OK]

Common Mistakes:

• Ignoring quantization for mobile
• Adding layers increases size and slows down
• Retraining large model after quantization wastes effort