Loading model state_dict in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of loading a state_dict with missing keys?

Consider a PyTorch model and a saved state_dict that lacks some keys present in the model. What happens when you load this state_dict with strict=True?

PyTorch

import torch
import torch.nn as nn

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(10, 5)
        self.fc2 = nn.Linear(5, 2)

model = SimpleModel()
saved_state = {'fc1.weight': torch.randn(5, 10), 'fc1.bias': torch.randn(5)}

try:
    model.load_state_dict(saved_state, strict=True)
    print('Loaded successfully')
except Exception as e:
    print(type(e).__name__)

A Loaded successfully

B KeyError

C RuntimeError

D TypeError

💡 Hint

Think about what strict=True means when keys are missing.

When loading a state_dict with strict=True, PyTorch expects all keys in the model to be present in the state_dict. Missing keys cause a RuntimeError.

❓ Model Choice

intermediate

Which option correctly loads a saved model state_dict ignoring missing keys?

You have a saved state_dict missing some keys. Which code snippet correctly loads it without error, ignoring missing keys?

A model.load_state_dict(saved_state, strict=False)

B model.load_state_dict(saved_state, strict=True)

C model.load_state_dict(saved_state, ignore_missing=True)

D model.load_state_dict(saved_state)

💡 Hint

Check the parameter that controls strict key matching.

Using strict=False allows loading a state_dict even if some keys are missing or extra, avoiding errors.

🔧 Debug

advanced

Why does this code raise a RuntimeError when loading a state_dict?

Examine the code below. Why does it raise a RuntimeError when loading the state_dict?

PyTorch

import torch
import torch.nn as nn

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer = nn.Linear(3, 2)

model = Model()
saved_state = {'layer.weight': torch.randn(2, 3), 'layer.bias': torch.randn(2), 'extra.weight': torch.randn(1)}

model.load_state_dict(saved_state)

A Because 'layer.weight' has wrong shape

B Because the model is not on the correct device

C Because 'layer.bias' is missing

D Because 'extra.weight' is not a key in the model's state_dict

💡 Hint

Look at keys in saved_state vs model's keys.

The saved_state contains an extra key 'extra.weight' which is not in the model's state_dict. By default, load_state_dict with strict=True raises a RuntimeError for unexpected keys.

❓ Hyperparameter

advanced

What does the strict parameter control in load_state_dict?

In PyTorch's load_state_dict method, what is the effect of setting strict=False?

A It forces the model to load only if all keys match exactly

B It allows loading state_dicts with missing or extra keys without error

C It loads the state_dict but resets optimizer parameters

D It converts all tensors to CPU before loading

💡 Hint

Think about key matching tolerance.

strict=False lets you load a state_dict even if some keys are missing or extra, avoiding errors.

🧠 Conceptual

expert

What is the best practice to load a state_dict when model architecture has changed?

You have updated your model architecture by adding new layers. You want to load weights from a previous checkpoint that lacks these new layers. What is the best practice to load the old weights safely?

A Load the state_dict with strict=False and manually initialize new layers

B Load the state_dict with strict=True to ensure all keys match

C Manually remove new layers from the model before loading state_dict

D Overwrite the model's state_dict with the checkpoint without loading

💡 Hint

Consider how to handle missing keys safely.

Using strict=False allows loading existing weights while ignoring new layers. Then you can initialize new layers separately.

Practice

1. What does model.load_state_dict() do in PyTorch?

easy

A. It loads saved model weights into the model.

B. It saves the current model weights to a file.

C. It initializes a new model architecture.

D. It compiles the model for training.

Solution

1. Step 1: Understand the purpose of load_state_dict

   This function is used to load previously saved weights into a model.

2. Step 2: Differentiate from other functions

   Saving weights uses state_dict() with torch.save(), not load_state_dict().

3. Final Answer:

   It loads saved model weights into the model. -> Option A

4. Quick Check:

   Load weights = load_state_dict() [OK]

Hint: Remember: load_state_dict loads weights, not saves them [OK]

Common Mistakes:

• Confusing loading weights with saving weights
• Thinking it initializes model architecture
• Assuming it compiles the model

2. Which of the following is the correct syntax to load a saved state dictionary from a file model.pth into a model named model?

easy

A. model.load_state_dict(torch.load('model.pth'))

B. model.load(torch.load_state_dict('model.pth'))

C. torch.load_state_dict(model, 'model.pth')

D. model.load_state_dict('model.pth')

Solution

1. Step 1: Identify correct function usage

   The correct way is to first load the saved weights with torch.load() and then pass them to model.load_state_dict().

2. Step 2: Check syntax correctness

   model.load_state_dict(torch.load('model.pth')) correctly calls torch.load('model.pth') inside model.load_state_dict(). Other options misuse function names or argument order.

3. Final Answer:

   model.load_state_dict(torch.load('model.pth')) -> Option A

4. Quick Check:

   Load weights with torch.load, then load_state_dict [OK]

Hint: Load file with torch.load, then pass to load_state_dict [OK]

Common Mistakes:

• Passing filename directly to load_state_dict
• Using wrong function names or order
• Confusing torch.load and load_state_dict

3. Given the code below, what will be printed?

import torch
import torch.nn as nn

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(2, 1)

model = SimpleModel()
torch.save(model.state_dict(), 'temp.pth')

new_model = SimpleModel()
new_model.load_state_dict(torch.load('temp.pth'))

print(all(torch.equal(p1, p2) for p1, p2 in zip(model.parameters(), new_model.parameters())))

medium

A. Raises an error

B. False

C. True

D. None

Solution

1. Step 1: Understand saving and loading state_dict

   The code saves the original model's weights and loads them into a new model instance.

2. Step 2: Compare parameters of both models

   Since the new model loads the exact saved weights, parameters should be identical, so the comparison returns True.

3. Final Answer:

   True -> Option C

4. Quick Check:

   Loaded weights match saved weights = True [OK]

Hint: Loaded model matches saved weights exactly [OK]

Common Mistakes:

• Assuming new model has random weights after loading
• Thinking load_state_dict changes model architecture
• Expecting an error due to missing device argument

4. You try to load a saved state_dict into your model but get this error: RuntimeError: Error(s) in loading state_dict for Model: Missing key(s) in state_dict: "fc.weight". What is the most likely cause?

medium

A. The file path to the saved state_dict is incorrect.

B. The saved state_dict is from a different model architecture.

C. You forgot to call torch.load() before loading.

D. The model was not moved to the correct device before loading.

Solution

1. Step 1: Analyze the error message

   The error says some keys are missing in the loaded state_dict, meaning the model expects parameters not found in the saved weights.

2. Step 2: Identify cause of missing keys

   This usually happens when the saved weights come from a different model architecture than the current model.

3. Final Answer:

   The saved state_dict is from a different model architecture. -> Option B

4. Quick Check:

   Missing keys = architecture mismatch [OK]

Hint: Missing keys usually mean model architectures differ [OK]

Common Mistakes:

• Assuming file path error causes missing keys
• Forgetting to load file before loading state_dict
• Thinking device mismatch causes missing keys

5. You have a model trained on GPU and saved its state_dict. Now you want to load it on a CPU-only machine. Which code snippet correctly loads the weights without error?

hard

A. model.load_state_dict(torch.load('model_gpu.pth', device='cpu'))

B. model.load_state_dict(torch.load('model_gpu.pth'))

C. model.load_state_dict(torch.load('model_gpu.pth', map_location='cuda'))

D. model.load_state_dict(torch.load('model_gpu.pth', map_location=torch.device('cpu')))

Solution

1. Step 1: Understand device mismatch issue

   Loading GPU-trained weights on CPU requires mapping the storage to CPU to avoid errors.

2. Step 2: Use correct map_location argument

   Passing map_location=torch.device('cpu') to torch.load() correctly maps tensors to CPU.

3. Final Answer:

   model.load_state_dict(torch.load('model_gpu.pth', map_location=torch.device('cpu'))) -> Option D

4. Quick Check:

   Use map_location to load GPU weights on CPU [OK]

Hint: Use map_location=torch.device('cpu') when loading GPU weights on CPU [OK]

Common Mistakes:

• Not using map_location causes runtime errors
• Passing wrong device string like 'cuda' on CPU
• Using non-existent 'device' argument in torch.load