Model optimization (quantization, pruning) in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Quantization Impact on Model Size

Which of the following statements best describes the main effect of applying post-training quantization to a PyTorch model?

A It increases the model size by adding extra layers to improve accuracy after training.

B It reduces the model size by converting weights from 32-bit floats to lower-bit integers, often 8-bit, with minimal accuracy loss.

C It removes neurons from the model to reduce size but always causes significant accuracy loss.

D It converts the model to use 64-bit floating point weights for higher precision.

💡 Hint

Think about how quantization changes the data type of weights and its effect on storage.

Quantization reduces model size by changing weights from 32-bit floats to smaller integer types like 8-bit integers, which saves memory and can speed up inference with little accuracy loss.

❓ Predict Output

intermediate

Output of Pruning Code on Model Parameters

Consider the following PyTorch code that applies pruning to a linear layer. What will be the number of zero weights in model.fc.weight after pruning?

PyTorch

import torch
import torch.nn as nn
import torch.nn.utils.prune as prune

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Linear(10, 5)

model = SimpleModel()
prune.l1_unstructured(model.fc, name='weight', amount=0.4)

zero_weights = torch.sum(model.fc.weight == 0).item()
print(zero_weights)

A 20

B 40

C 12

D 0

💡 Hint

Calculate total weights and 40% of them.

The linear layer has 10 inputs and 5 outputs, so 50 weights total. 40% of 50 is 20, so 20 weights are zeroed by pruning.

❓ Hyperparameter

advanced

Choosing Pruning Amount for Accuracy Preservation

You want to prune a PyTorch model to reduce size but keep accuracy loss under 2%. Which pruning amount is most likely to meet this goal?

A Pruning 5% of weights using L1 unstructured pruning

B Pruning 10% of weights using structured pruning on entire channels

C Pruning 50% of weights randomly without considering importance

D Pruning 80% of weights using unstructured pruning

💡 Hint

Smaller pruning amounts usually preserve accuracy better.

Pruning only 5% of weights with L1 unstructured pruning removes the least important weights, minimizing accuracy loss under 2%. Larger amounts or random pruning risk higher loss.

❓ Metrics

advanced

Evaluating Quantized Model Accuracy

After applying dynamic quantization to a PyTorch LSTM model, you observe the following accuracies on the test set:

• Original model accuracy: 92%
• Quantized model accuracy: 89%

What is the best interpretation of this result?

A The quantized model should have the same accuracy; the difference means the test data changed.

B Dynamic quantization always improves accuracy, so this result is unexpected and indicates a bug.

C Quantization caused the model to overfit, increasing accuracy on training but decreasing on test.

D Dynamic quantization caused a small accuracy drop but improved inference speed and reduced model size.

💡 Hint

Quantization trades off some accuracy for efficiency.

Dynamic quantization reduces model size and speeds up inference but can cause a small accuracy drop, which is normal and expected.

🔧 Debug

expert

Identifying Error in Pruning Application Code

What error will the following PyTorch code raise when trying to prune a model's convolutional layer?

PyTorch

import torch
import torch.nn as nn
import torch.nn.utils.prune as prune

class ConvModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(3, 16, 3, bias=False)

model = ConvModel()
prune.l1_unstructured(model.conv, name='bias', amount=0.3)

A No error; pruning applied successfully

B AttributeError: 'Conv2d' object has no attribute 'bias_mask'

C ValueError: Cannot prune parameter 'bias' because it does not exist or is not a tensor

D RuntimeError: Pruning amount must be between 0 and 1

💡 Hint

Check if the layer has a bias parameter by default.

The Conv2d layer by default has a bias parameter, but if bias=False is set, it does not. If bias is None, pruning 'bias' raises ValueError because the parameter does not exist.

Practice

1. What is the main goal of model pruning in PyTorch?

easy

A. Add more training data to improve accuracy

B. Increase the number of layers in the model

C. Change the model architecture to a different type

D. Remove less important weights to make the model smaller

Solution

1. Step 1: Understand pruning concept

   Pruning means removing weights that do not contribute much to the model's output.

2. Step 2: Identify pruning effect

   Removing these weights makes the model smaller and faster without changing its structure.

3. Final Answer:

   Remove less important weights to make the model smaller -> Option D

4. Quick Check:

   Pruning = Remove less important weights [OK]

Hint: Pruning cuts unimportant weights to shrink model size [OK]

Common Mistakes:

• Thinking pruning adds layers
• Confusing pruning with data augmentation
• Believing pruning changes model type

2. Which PyTorch function is used to apply pruning to a layer?

easy

A. torch.nn.utils.prune.l1_unstructured

B. torch.quantization.quantize_dynamic

C. torch.optim.Adam

D. torch.nn.functional.relu

Solution

1. Step 1: Identify pruning functions in PyTorch

   PyTorch pruning utilities are in torch.nn.utils.prune module.

2. Step 2: Match pruning function

   l1_unstructured is a common pruning method that removes weights based on L1 norm.

3. Final Answer:

   torch.nn.utils.prune.l1_unstructured -> Option A

4. Quick Check:

   Pruning function = l1_unstructured [OK]

Hint: Pruning uses torch.nn.utils.prune functions [OK]

Common Mistakes:

• Confusing pruning with quantization functions
• Choosing optimizer functions by mistake
• Selecting activation functions instead of pruning

3. What will be the output shape of the model's weight tensor after applying this pruning code?

import torch
import torch.nn.utils.prune as prune

layer = torch.nn.Linear(4, 3)
prune.l1_unstructured(layer, name='weight', amount=0.5)
print(layer.weight.shape)

medium

A. torch.Size([4, 3]) with half weights removed

B. torch.Size([3, 4]) with half weights removed physically

C. torch.Size([3, 4]) with some weights zeroed

D. torch.Size([4, 3]) unchanged

Solution

1. Step 1: Check layer weight shape

   A Linear layer with input 4 and output 3 has weight shape (3, 4).

2. Step 2: Understand pruning effect on weight tensor

   Pruning sets some weights to zero but does not change tensor shape.

3. Final Answer:

   torch.Size([3, 4]) with some weights zeroed -> Option C

4. Quick Check:

   Pruning zeros weights, shape stays same [OK]

Hint: Pruning zeros weights but keeps tensor shape [OK]

Common Mistakes:

• Thinking pruning changes tensor shape
• Confusing input/output dimensions order
• Assuming weights are physically removed

4. Identify the error in this PyTorch pruning code:

import torch.nn.utils.prune as prune
layer = torch.nn.Linear(5, 2)
prune.l1_unstructured(layer, name='bias', amount=0.3)

medium

A. Missing import for torch

B. Layer input and output sizes are invalid

C. Amount must be an integer, not float

D. Cannot prune bias with l1_unstructured

Solution

1. Step 1: Examine the code imports

   The code imports only prune but attempts to use torch.nn.Linear without importing torch.

2. Step 2: Identify the runtime error

   This causes NameError: name 'torch' is not defined when creating the layer.

3. Final Answer:

   Missing import for torch -> Option A

4. Quick Check:

   Missing torch import causes NameError [OK]

Hint: Import torch before using torch.nn [OK]

Common Mistakes:

• Overlooking missing torch import
• Thinking pruning bias is unsupported
• Assuming amount must be integer

5. You want to speed up inference on a PyTorch model by reducing number precision. Which approach should you use?

hard

A. Increase batch size during training

B. Apply dynamic quantization to convert weights to lower precision

C. Prune 90% of the model weights randomly

D. Add dropout layers to the model

Solution

1. Step 1: Understand quantization purpose

   Quantization reduces number precision (e.g., float32 to int8) to speed up inference.

2. Step 2: Identify correct method

   Dynamic quantization in PyTorch converts weights to lower precision without retraining.

3. Step 3: Compare other options

   Pruning removes weights but may not speed inference as much; batch size and dropout affect training, not inference speed.

4. Final Answer:

   Apply dynamic quantization to convert weights to lower precision -> Option B

5. Quick Check:

   Quantization = lower precision for speed [OK]

Hint: Quantization reduces precision to speed inference [OK]

Common Mistakes:

• Confusing pruning with quantization
• Thinking batch size affects model speed directly
• Adding dropout to speed inference