Model optimization (distillation, quantization) in NLP - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to load a pre-trained model for distillation.

NLP

from transformers import DistilBertForSequenceClassification
model = DistilBertForSequenceClassification.from_pretrained([1])

Drag options to blanks, or click blank then click option'

A"distilbert-base-uncased"

B"bert-base-uncased"

C"gpt2"

D"roberta-base"

Attempts:

3 left

💡 Hint

Common Mistakes

Using the full BERT model name instead of the distilled version.

Choosing a model from a different architecture like GPT-2.

DistilBERT is a smaller, distilled version of BERT, so the correct model name is "distilbert-base-uncased".

2fill in blank

medium

Complete the code to apply dynamic quantization to a PyTorch model.

NLP

import torch
model = torch.quantization.quantize_dynamic(model, [1], dtype=torch.qint8)

Drag options to blanks, or click blank then click option'

A[torch.nn.LSTM]

B[torch.nn.Linear]

C[torch.nn.Conv2d]

D[torch.nn.ReLU]

Attempts:

3 left

💡 Hint

Common Mistakes

Trying to quantize activation functions like ReLU.

Using convolution layers which are less common in NLP models.

Dynamic quantization is commonly applied to linear layers to reduce model size and speed up inference.

3fill in blank

hard

Fix the error in the code to correctly perform knowledge distillation training.

NLP

teacher_outputs = teacher_model(input_ids)
student_outputs = student_model(input_ids)
loss = distillation_loss(student_outputs, teacher_outputs[1])

Drag options to blanks, or click blank then click option'

A.attentions

B.hidden_states

C.logits

D.labels

Attempts:

3 left

💡 Hint

Common Mistakes

Using hidden states or attention outputs instead of logits.

Trying to access labels from model outputs.

The logits are the raw predictions needed to compute the distillation loss between teacher and student.

4fill in blank

hard

Fill both blanks to create a quantized model and prepare it for inference.

NLP

import torch.quantization
quantized_model = torch.quantization.quantize_dynamic(model, [1], dtype=[2])

Drag options to blanks, or click blank then click option'

A[torch.nn.Linear]

Btorch.qint8

Ctorch.float16

D[torch.nn.Conv2d]

Attempts:

3 left

💡 Hint

Common Mistakes

Using float16 which is not dynamic quantization dtype.

Trying to quantize convolution layers in NLP models.

Dynamic quantization targets Linear layers and uses qint8 data type for quantization.

5fill in blank

hard

Fill all three blanks to define a distillation loss combining student and teacher outputs.

NLP

import torch.nn.functional as F
alpha = 0.5
T = 2.0
loss = alpha * F.kl_div(F.log_softmax(student_outputs[1] / T, dim=1), F.softmax(teacher_outputs[2] / T, dim=1), reduction='batchmean') * (T * T) + (1 - alpha) * F.cross_entropy(student_outputs[3], labels)

Drag options to blanks, or click blank then click option'

A.logits

D.hidden_states

Attempts:

3 left

💡 Hint

Common Mistakes

Using hidden states instead of logits.

Mixing up student and teacher outputs.

The distillation loss uses logits from both student and teacher models for KL divergence and student logits for cross-entropy with true labels.

Practice

1. What is the main goal of model distillation in NLP?

easy

A. To increase the number of layers in a neural network

B. To add more training data for better accuracy

C. To convert text data into numerical vectors

D. To train a smaller model to mimic a larger model's behavior

Solution

1. Step 1: Understand model distillation concept

   Model distillation is about making a smaller model learn from a bigger, well-trained model.

2. Step 2: Identify the goal of distillation

   The goal is to keep performance while reducing model size and complexity.

3. Final Answer:

   To train a smaller model to mimic a larger model's behavior -> Option D

4. Quick Check:

   Distillation = smaller model copies bigger model [OK]

Hint: Distillation means small model learns from big model [OK]

Common Mistakes:

• Confusing distillation with adding layers
• Thinking distillation increases data size
• Mixing distillation with data preprocessing

2. Which of the following is the correct way to apply quantization to a model's weights in Python using PyTorch?

easy

A. model.quantize(weights=True)

B. torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)

C. torch.quantize(model, dtype=torch.float32)

D. torch.quantization(model, dtype=torch.int32)

Solution

1. Step 1: Recall PyTorch quantization syntax

   PyTorch uses torch.quantization.quantize_dynamic for dynamic quantization on layers like Linear.

2. Step 2: Check correct function and parameters

   torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8) correctly calls quantize_dynamic with model, target layers, and dtype torch.qint8.

3. Final Answer:

   torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8) -> Option B

4. Quick Check:

   PyTorch quantize_dynamic with Linear and qint8 = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8) [OK]

Hint: Use torch.quantization.quantize_dynamic for quantization [OK]

Common Mistakes:

• Using non-existent torch.quantize function
• Passing wrong dtype like float32 instead of qint8
• Calling quantization as a model method

3. Given the following code snippet for distillation, what will be the output loss value if the student model perfectly mimics the teacher model's outputs?

teacher_outputs = torch.tensor([0.1, 0.9])
student_outputs = torch.tensor([0.1, 0.9])
loss_fn = torch.nn.MSELoss()
loss = loss_fn(student_outputs, teacher_outputs)
print(loss.item())

medium

A. 0.0

B. 0.5

C. 1.0

D. Cannot compute due to shape mismatch

Solution

1. Step 1: Understand MSELoss calculation

   MSELoss calculates mean squared error between student and teacher outputs.

2. Step 2: Calculate loss for identical outputs

   Since student_outputs equals teacher_outputs, difference is zero, so loss is 0.0.

3. Final Answer:

   0.0 -> Option A

4. Quick Check:

   Identical outputs give zero MSE loss [OK]

Hint: Same outputs mean zero loss in MSE [OK]

Common Mistakes:

• Assuming loss is 1.0 by default
• Confusing loss with accuracy
• Thinking shape mismatch error occurs

4. You tried to quantize a model but got an error: AttributeError: 'MyModel' object has no attribute 'quantize'. What is the likely cause?

medium

A. The model class does not have a built-in quantize method

B. You forgot to import torch

C. Quantization only works on CPU, not GPU

D. The model is already quantized

Solution

1. Step 1: Analyze the error message

   The error says the model object lacks a 'quantize' method, meaning it is not defined.

2. Step 2: Understand quantization usage

   Quantization is applied via PyTorch functions, not as a model method, so calling model.quantize() causes error.

3. Final Answer:

   The model class does not have a built-in quantize method -> Option A

4. Quick Check:

   Quantize is a function, not a model method [OK]

Hint: Quantize via torch functions, not model methods [OK]

Common Mistakes:

• Trying to call quantize as model.quantize()
• Ignoring import errors
• Assuming quantization only works on CPU

5. You want to deploy a chatbot on a mobile device with limited memory and CPU. Which combination of model optimization techniques is best to reduce size and speed up inference without losing much accuracy?

hard

A. Use quantization first, then retrain the large model from scratch

B. Only increase the training data size to improve accuracy

C. Use distillation to train a smaller model, then apply quantization to reduce precision

D. Add more layers to the model and use float64 precision

Solution

1. Step 1: Identify constraints and goals

   Mobile devices need small, fast models with good accuracy.

2. Step 2: Choose suitable optimization techniques

   Distillation creates a smaller model; quantization reduces number precision to save space and speed up inference.

3. Step 3: Combine techniques for best effect

   Using distillation first then quantization is a common, effective approach.

4. Final Answer:

   Use distillation to train a smaller model, then apply quantization to reduce precision -> Option C

5. Quick Check:

   Distillation + quantization = small, fast, accurate model [OK]

Hint: Distill first, then quantize for mobile deployment [OK]

Common Mistakes:

• Ignoring quantization for mobile
• Adding layers increases size and slows down
• Retraining large model after quantization wastes effort