DataParallel basics in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why use DataParallel in PyTorch?

You have a deep learning model and a machine with multiple GPUs. What is the main reason to use DataParallel in PyTorch?

A To reduce the size of the model so it fits into a single GPU memory.

B To split the input data across multiple GPUs and run the model in parallel to speed up training.

C To automatically tune hyperparameters during training.

D To convert the model into a CPU-only version for faster inference.

💡 Hint

Think about how multiple GPUs can be used to handle more data at once.

DataParallel splits the input batch across multiple GPUs, runs the model on each part in parallel, and then combines the results. This speeds up training by using all GPUs efficiently.

❓ Predict Output

intermediate

Output of DataParallel model forward pass

Consider this PyTorch code snippet using DataParallel:

PyTorch

import torch
import torch.nn as nn

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(2, 1)
    def forward(self, x):
        return self.linear(x)

model = SimpleModel()
model = nn.DataParallel(model)
input_tensor = torch.tensor([[1.0, 2.0], [3.0, 4.0]])
output = model(input_tensor)
print(output.shape)

A RuntimeError due to input on CPU but model on GPU

B torch.Size([1, 2])

C torch.Size([2, 1])

D torch.Size([4, 1])

💡 Hint

DataParallel splits the batch dimension but output shape matches input batch size.

The input has batch size 2 and feature size 2. The linear layer outputs size 1 per input. DataParallel splits the batch but output shape remains (2,1).

❓ Hyperparameter

advanced

Choosing batch size with DataParallel

You want to train a model using DataParallel on 4 GPUs. Your original batch size is 64. What is the best way to set the batch size when using DataParallel?

A Keep batch size 64; DataParallel will split it automatically across GPUs.

B Set batch size to 16 because DataParallel requires batch size per GPU.

C Increase batch size to 256 to fully use all GPUs.

D Set batch size to 1 because DataParallel only supports batch size 1.

💡 Hint

Think about how DataParallel handles the batch dimension internally.

DataParallel automatically splits the batch across GPUs. So you can keep the original batch size, and it will divide it evenly among GPUs.

🔧 Debug

advanced

Why does this DataParallel code raise an error?

Look at this code snippet:

PyTorch

import torch
import torch.nn as nn

model = nn.Linear(10, 5)
model = nn.DataParallel(model)
input_tensor = torch.randn(3, 10).cuda()
output = model(input_tensor)
print(output)

A Raises AttributeError because DataParallel is not imported.

B Runs fine and prints output tensor.

C Raises TypeError because input tensor shape is wrong.

D Raises RuntimeError because model is on CPU but input is on GPU.

💡 Hint

Check where the model parameters are located before wrapping with DataParallel.

The model is created on CPU by default. DataParallel expects the model to be on GPU before wrapping. Input is on GPU, so mismatch causes RuntimeError.

❓ Model Choice

expert

Best model wrapping for multi-GPU training

You have a complex model with custom layers and want to train on multiple GPUs. Which approach is best to ensure correct gradient updates and efficient training?

A Use nn.parallel.DistributedDataParallel with proper setup for multi-GPU training.

B Train the model on CPU and manually split data across GPUs.

C Wrap the model with nn.DataParallel before moving it to GPU.

D Wrap the model with nn.DataParallel after moving it to GPU.

💡 Hint

Consider scalability and correctness for multi-GPU training.

DistributedDataParallel is recommended for multi-GPU training because it handles gradient synchronization efficiently and avoids common pitfalls of DataParallel.

Practice

1. What is the main purpose of torch.nn.DataParallel in PyTorch?

easy

A. To convert a model to run only on CPU

B. To run a model on multiple GPUs by splitting input data automatically

C. To reduce the size of the model parameters

D. To change the model architecture dynamically during training

Solution

1. Step 1: Understand DataParallel's role

   DataParallel helps use multiple GPUs by splitting input data and running model copies in parallel.

2. Step 2: Compare options

   Only To run a model on multiple GPUs by splitting input data automatically describes this behavior correctly; others describe unrelated tasks.

3. Final Answer:

   To run a model on multiple GPUs by splitting input data automatically -> Option B

4. Quick Check:

   DataParallel = multi-GPU input splitting [OK]

Hint: DataParallel splits input across GPUs automatically [OK]

Common Mistakes:

• Thinking DataParallel changes model architecture
• Confusing DataParallel with CPU-only mode
• Believing it reduces model size

2. Which of the following is the correct way to wrap a PyTorch model model with DataParallel?

easy

A. model = torch.DataParallel(model)

B. model = torch.nn.parallel.DataParallel(model)

C. model = torch.nn.DataParallel(model)

D. model = torch.parallel.DataParallel(model)

Solution

1. Step 1: Recall correct import path

   DataParallel is in torch.nn, so correct usage is torch.nn.DataParallel.

2. Step 2: Check options

   model = torch.nn.DataParallel(model) matches the correct usage; others are incorrect or incomplete.

3. Final Answer:

   model = torch.nn.DataParallel(model) -> Option C

4. Quick Check:

   Correct import path = torch.nn.DataParallel [OK]

Hint: DataParallel is in torch.nn module [OK]

Common Mistakes:

• Using torch.DataParallel instead of torch.nn.DataParallel
• Confusing torch.nn.parallel with torch.nn
• Confusing with other torch submodules

3. Given the code below, what will be the output shape of output if input shape is (8, 3, 32, 32) and 2 GPUs are used?

import torch
import torch.nn as nn

model = nn.Linear(32*32*3, 10)
model = nn.parallel.DataParallel(model)
input = torch.randn(8, 3, 32, 32)
input = input.view(8, -1)
output = model(input)
print(output.shape)

medium

A. torch.Size([8, 10])

B. torch.Size([4, 10])

C. torch.Size([16, 10])

D. RuntimeError due to shape mismatch

Solution

1. Step 1: Understand input and model

   Input is reshaped to (8, 3072) and model outputs 10 features per input.

2. Step 2: DataParallel splits batch across 2 GPUs

   Batch of 8 splits into 2 batches of 4, processed in parallel, outputs combined back to (8, 10).

3. Final Answer:

   torch.Size([8, 10]) -> Option A

4. Quick Check:

   Output shape = batch size x output features = (8, 10) [OK]

Hint: Output batch size stays same after DataParallel [OK]

Common Mistakes:

• Thinking output batch size halves with GPUs
• Confusing input reshape with output shape
• Expecting runtime error due to DataParallel

4. What is the likely cause of this error when using DataParallel?

RuntimeError: Expected tensor for argument #1 'input' to have scalar type Float but got Double

Code snippet:

model = torch.nn.parallel.DataParallel(model)
input = torch.randn(8, 3, 32, 32, dtype=torch.double)
output = model(input)

medium

A. Input batch size is too large for GPUs

B. DataParallel requires input to be on CPU only

C. Model is not wrapped correctly with DataParallel

D. Input tensor dtype does not match model parameters dtype

Solution

1. Step 1: Analyze error message

   Error says input tensor is Double but model expects Float tensors.

2. Step 2: Check input and model dtypes

   Model parameters default to Float; input is Double, causing mismatch and error.

3. Final Answer:

   Input tensor dtype does not match model parameters dtype -> Option D

4. Quick Check:

   Dtype mismatch causes runtime error [OK]

Hint: Ensure input and model dtypes match before DataParallel [OK]

Common Mistakes:

• Assuming DataParallel only works on CPU
• Not checking tensor data types
• Thinking batch size causes dtype errors

5. You want to speed up training on 4 GPUs using DataParallel. Your model has a batch normalization layer. What should you be careful about to avoid incorrect training behavior?

hard

A. BatchNorm layers behave differently in DataParallel; use SyncBatchNorm or move to DistributedDataParallel

B. DataParallel automatically fixes BatchNorm issues, no action needed

C. Remove BatchNorm layers before using DataParallel

D. Increase batch size to avoid BatchNorm problems

Solution

1. Step 1: Understand BatchNorm behavior in DataParallel

   BatchNorm computes stats per device, which can cause inconsistent normalization across GPUs.

2. Step 2: Recommended solutions

   Use SyncBatchNorm to sync stats or switch to DistributedDataParallel for correct BatchNorm behavior.

3. Final Answer:

   BatchNorm layers behave differently in DataParallel; use SyncBatchNorm or move to DistributedDataParallel -> Option A

4. Quick Check:

   BatchNorm needs syncing or DDP for multi-GPU [OK]

Hint: BatchNorm needs syncing or DDP on multiple GPUs [OK]

Common Mistakes:

• Assuming DataParallel handles BatchNorm correctly by default
• Removing BatchNorm layers unnecessarily
• Only increasing batch size without syncing stats