Feature extraction strategy in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of feature extraction with frozen layers

Consider a pretrained ResNet18 model in PyTorch. You freeze all layers except the last fully connected layer and pass a batch of images through it. What will be the shape of the output tensor?

PyTorch

import torch
import torchvision.models as models

model = models.resnet18(pretrained=True)
for param in model.parameters():
    param.requires_grad = False
model.fc = torch.nn.Linear(model.fc.in_features, 10)  # new output classes

inputs = torch.randn(8, 3, 224, 224)  # batch of 8 images
outputs = model(inputs)
print(outputs.shape)

A torch.Size([1, 10])

B torch.Size([8, 512])

C torch.Size([8, 1000])

D torch.Size([8, 10])

💡 Hint

Think about the batch size and the number of output classes after replacing the last layer.

The model outputs a tensor with shape (batch_size, number_of_classes). Since batch size is 8 and the last layer outputs 10 classes, the shape is (8, 10).

❓ Model Choice

intermediate

Best model choice for feature extraction on small dataset

You want to use feature extraction on a small image dataset with limited labels. Which pretrained model is generally best to start with for extracting features?

A A small pretrained MobileNetV2 model

B A pretrained VGG16 model

C A large pretrained ResNet50 model

D A pretrained Transformer-based model like ViT

💡 Hint

Consider model size and overfitting risk on small datasets.

MobileNetV2 is lightweight and less likely to overfit on small datasets, making it a good choice for feature extraction in such cases.

❓ Hyperparameter

advanced

Choosing learning rate for fine-tuning after feature extraction

After extracting features using a pretrained model and replacing the last layer, you want to fine-tune the model. Which learning rate setting is most appropriate to start with?

A A very high learning rate like 0.1

B A moderate learning rate like 0.01

C A very low learning rate like 1e-5

D No learning rate, freeze all layers

💡 Hint

Fine-tuning pretrained weights requires careful small updates.

A very low learning rate like 1e-5 helps fine-tune pretrained weights gently without destroying learned features.

❓ Metrics

advanced

Evaluating feature extraction effectiveness

You extracted features from a pretrained model and trained a classifier on top. Which metric best shows if feature extraction helped improve classification?

A Validation accuracy compared to training from scratch

B Training loss of the classifier

C Number of model parameters

D Time taken to train the classifier

💡 Hint

Think about comparing performance with and without feature extraction.

Validation accuracy compared to training from scratch shows if feature extraction improved generalization.

🔧 Debug

expert

Debugging feature extraction output mismatch

You extract features from a pretrained CNN by removing the last layer. But your extracted features have shape (8, 512, 1, 1) instead of expected (8, 512). What is the likely cause?

PyTorch

import torch
import torchvision.models as models

model = models.resnet18(pretrained=True)
# Attempt to remove last layer
model = torch.nn.Sequential(*list(model.children())[:-1])

inputs = torch.randn(8, 3, 224, 224)
features = model(inputs)
print(features.shape)

A The last layer was not removed properly; output is from the final FC layer

B The model output is flattened incorrectly, so shape is wrong

C Input batch size is wrong causing unexpected output shape

D The model expects different input image size

💡 Hint

Check if the output tensor is still 4D and needs flattening.

Removing the last layer returns a 4D tensor (batch, channels, 1, 1). Without flattening, shape appears as (8, 512, 1, 1). Proper flattening is needed.

Practice

1. What is the main purpose of using a pre-trained model for feature extraction in PyTorch?

easy

A. To replace the optimizer with a new one

B. To use learned features from a large dataset and avoid training from scratch

C. To train all layers from random weights

D. To increase the size of the dataset automatically

Solution

1. Step 1: Understand feature extraction concept

   Feature extraction uses a model already trained on a large dataset to get useful features without training all layers again.

2. Step 2: Identify the main benefit

   This saves time and resources by reusing learned knowledge instead of starting from scratch.

3. Final Answer:

   To use learned features from a large dataset and avoid training from scratch -> Option B

4. Quick Check:

   Feature extraction = reuse learned features [OK]

Hint: Pre-trained means reuse, not retrain all layers [OK]

Common Mistakes:

• Thinking feature extraction means training all layers
• Confusing feature extraction with data augmentation
• Believing optimizer changes are part of feature extraction

2. Which PyTorch code snippet correctly freezes all layers of a pre-trained model except the final layer?

easy

A. for param in model.parameters(): param.requires_grad = True
model.fc = nn.Linear(512, 10)

B. model.fc.requires_grad = False
for param in model.parameters(): param.requires_grad = True

C. for param in model.parameters(): param.requires_grad = False
model.fc = nn.Linear(512, 10)

D. model.fc = nn.Linear(512, 10)
for param in model.parameters(): param.requires_grad = False

Solution

1. Step 1: Freeze all layers by setting requires_grad to false

   The loop disables gradient updates for all parameters to keep pre-trained weights fixed.

2. Step 2: Replace the final layer with a new one to train

   Assigning a new linear layer to model.fc allows training only this layer for the new task.

3. Final Answer:

   for param in model.parameters(): param.requires_grad = False model.fc = nn.Linear(512, 10) -> Option C

4. Quick Check:

   Freeze all except final layer = for param in model.parameters(): param.requires_grad = False model.fc = nn.Linear(512, 10) [OK]

Hint: Freeze first, then replace final layer [OK]

Common Mistakes:

• Not freezing layers before replacing final layer
• Freezing final layer instead of others
• Setting requires_grad true for all parameters

3. Given this PyTorch code for feature extraction, what will be the output shape of features?

import torch
import torchvision.models as models
model = models.resnet18(pretrained=True)
model.fc = torch.nn.Identity()
input_tensor = torch.randn(4, 3, 224, 224)
features = model(input_tensor)
print(features.shape)

medium

A. torch.Size([4, 512])

B. torch.Size([4, 1000])

C. torch.Size([4, 3, 224, 224])

D. torch.Size([4, 2048])

Solution

1. Step 1: Understand model modification

   Replacing model.fc with Identity removes the final classification layer, so output is the feature vector before classification.

2. Step 2: Know ResNet18 feature size

   ResNet18 outputs a 512-dimensional vector before the final fc layer for each input image.

3. Final Answer:

   torch.Size([4, 512]) -> Option A

4. Quick Check:

   ResNet18 features = 512 dims [OK]

Hint: Identity layer outputs feature vector size [OK]

Common Mistakes:

• Assuming output is 1000 classes without removing fc
• Confusing batch size with feature dimension
• Expecting 2048 features from ResNet18 (it's 512)

4. Identify the error in this feature extraction code snippet and select the fix:

model = models.resnet50(pretrained=True)
for param in model.parameters():
    param.requires_grad = False
model.fc = nn.Linear(2048, 5)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

# Training loop here

medium

A. No error; code is correct

B. Set requires_grad=True for model.fc parameters after replacement

C. Use Adam optimizer instead of SGD

D. Remove freezing of parameters to train all layers

Solution

1. Step 1: Check freezing timing

   The loop freezes existing parameters before replacing model.fc, so the new fc layer's parameters are created with requires_grad=True by default.

2. Step 2: Verify optimizer behavior

   Optimizer only updates parameters where requires_grad=True, which are the new fc parameters; backbone remains frozen.

3. Final Answer:

   No error; code is correct -> Option A

4. Quick Check:

   New layer params unfrozen by default [OK]

Hint: New layers have requires_grad=True by default [OK]

Common Mistakes:

• Assuming freezing all parameters includes new layers
• Changing optimizer without fixing requires_grad
• Removing freezing unnecessarily

5. You want to use a pre-trained ResNet34 to classify 3 classes in your dataset. You freeze all layers except the last one. However, your training accuracy stays very low. What is the best next step to improve feature extraction performance?

hard

A. Reduce batch size to 1 to improve gradient estimates

B. Increase learning rate to 1.0 for faster training

C. Replace the optimizer with SGD without momentum

D. Unfreeze some deeper layers to fine-tune features for your task

Solution

1. Step 1: Understand freezing impact

   Freezing all but last layer may limit model's ability to adapt features to new classes, causing low accuracy.

2. Step 2: Fine-tune some deeper layers

   Unfreezing some layers closer to output allows the model to adjust features better for your specific dataset.

3. Final Answer:

   Unfreeze some deeper layers to fine-tune features for your task -> Option D

4. Quick Check:

   Fine-tune layers = better adaptation [OK]

Hint: Fine-tune layers if frozen model underperforms [OK]

Common Mistakes:

• Increasing learning rate too much causes instability
• Changing optimizer without addressing feature adaptation
• Reducing batch size unnecessarily