Model optimization (distillation, quantization) in NLP - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Model Distillation Purpose

What is the main goal of model distillation in NLP?

A To train a smaller model to mimic a larger model's behavior

B To increase the number of parameters in the model for better accuracy

C To add noise to the training data to improve robustness

D To convert a model into a rule-based system

💡 Hint

Think about how a big model can help a smaller one learn.

Model distillation trains a smaller model (student) to imitate a larger, well-trained model (teacher), keeping performance but reducing size.

❓ Predict Output

intermediate

Output of Quantization Code Snippet

What is the output shape of the quantized model's embedding layer weights after applying 8-bit quantization?

NLP

import torch
import torch.nn as nn

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding = nn.Embedding(1000, 64)

model = SimpleModel()
quantized_model = torch.quantization.quantize_dynamic(
    model, {nn.Embedding}, dtype=torch.qint8
)
weight_shape = quantized_model.embedding.weight.shape
print(weight_shape)

A AttributeError

B (64, 1000)

C (1000, 32)

D (1000, 64)

💡 Hint

Quantization changes data type but not tensor shape.

The quantized embedding weights keep the original shape (1000, 64). Quantization changes precision, not dimensions.

❓ Model Choice

advanced

Choosing a Model for Distillation

You want to distill a large BERT model into a smaller one for mobile deployment. Which student model architecture is best suited?

A A recurrent neural network with LSTM cells

B A convolutional neural network designed for images

C A smaller BERT model with fewer layers and parameters

D A large transformer model with more layers than the teacher

💡 Hint

Student model should be similar but smaller than the teacher.

Distillation works best when the student model architecture is similar but smaller, like a smaller BERT variant, to capture the teacher's knowledge efficiently.

❓ Hyperparameter

advanced

Key Hyperparameter in Quantization

Which hyperparameter is critical to set correctly when applying post-training quantization to an NLP model?

A The learning rate of the optimizer

B The number of bits used to represent weights and activations

C The dropout rate in the model layers

D The batch size during training

💡 Hint

Quantization precision depends on this setting.

The bit-width (e.g., 8-bit) determines how finely weights and activations are represented, affecting model size and accuracy.

❓ Metrics

expert

Evaluating Distilled Model Performance

After distilling a large NLP model, which metric best shows if the smaller model retained the teacher's knowledge effectively?

A Accuracy on a held-out validation set compared to the teacher model

B Training loss of the student model on the training data

C Number of parameters in the student model

D Inference time on a GPU

💡 Hint

Think about measuring how well the student predicts compared to the teacher.

Accuracy on validation data shows if the student model learned to mimic the teacher's predictions well, indicating successful distillation.