Multi-GPU training in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of PyTorch DataParallel model forward pass

Consider the following PyTorch code snippet using DataParallel for multi-GPU training. What will be the output shape of output after the forward pass?

PyTorch

import torch
import torch.nn as nn

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(10, 5)

    def forward(self, x):
        return self.linear(x)

model = SimpleModel()
if torch.cuda.device_count() > 1:
    model = nn.DataParallel(model)

model = model.cuda()
input_tensor = torch.randn(16, 10).cuda()
output = model(input_tensor)
print(output.shape)

A torch.Size([8, 5])

B torch.Size([32, 5])

C RuntimeError due to input size mismatch

D torch.Size([16, 5])

💡 Hint

DataParallel splits the batch across GPUs but concatenates outputs back to original batch size.

DataParallel splits the input batch (size 16) across available GPUs, processes them in parallel, then concatenates the outputs. The output shape matches the input batch size with the output feature size (5).

🧠 Conceptual

intermediate

Understanding DistributedDataParallel vs DataParallel

Which statement correctly describes a key difference between PyTorch's DistributedDataParallel (DDP) and DataParallel (DP) for multi-GPU training?

A DP requires manual gradient synchronization, but DDP automatically splits input batches across GPUs.

B DDP runs model replicas in separate processes and synchronizes gradients, while DP runs replicas in a single process splitting input batches.

C DDP only works on a single GPU, while DP supports multiple GPUs on one machine.

D DP uses multiple machines for training, while DDP is limited to one machine.

💡 Hint

Think about process management and how input batches are handled.

DistributedDataParallel runs one process per GPU and synchronizes gradients across processes, which is more efficient and scalable. DataParallel runs in one process and splits input batches across GPUs, which can be slower.

❓ Hyperparameter

advanced

Choosing batch size for multi-GPU training

You want to train a model on 4 GPUs using DistributedDataParallel. Your single-GPU batch size is 32. What is the best practice for setting the batch size when using 4 GPUs?

A Keep batch size 32 and let each GPU process the full batch independently.

B Set batch size to 8 so total batch size is 8 across all GPUs.

C Set batch size to 128 so each GPU processes 32 samples (total batch size = 4 * 32).

D Set batch size to 32 and manually average gradients across GPUs.

💡 Hint

Consider how DistributedDataParallel splits data across GPUs.

In DDP, each GPU processes a subset of the batch. To keep the effective batch size the same as single-GPU training, multiply the batch size by the number of GPUs.

🔧 Debug

advanced

Identifying error in multi-GPU model saving

You trained a model using nn.DataParallel and saved it with torch.save(model.state_dict(), 'model.pth'). When loading the model later without DataParallel, you get a key mismatch error. What is the cause?

A The saved state dict keys are prefixed with 'module.' due to DataParallel wrapping.

B The model was saved on CPU but loaded on GPU causing mismatch.

C The model architecture changed between saving and loading causing mismatch.

D The saved file is corrupted and cannot be loaded properly.

💡 Hint

DataParallel adds a prefix to parameter names in the state dict.

DataParallel wraps the model and prefixes all parameter keys with 'module.'. Loading this state dict into a model without DataParallel causes key mismatch errors.

❓ Metrics

expert

Interpreting training speedup with multi-GPU training

You train a model on 1 GPU and it takes 100 minutes per epoch. Using 4 GPUs with DistributedDataParallel, the epoch time reduces to 30 minutes. Which statement best explains this result?

A The speedup is less than 4x due to communication overhead and synchronization costs.

B The speedup is exactly 4x because 4 GPUs process data independently without overhead.

C The speedup is more than 4x because GPUs work better together than alone.

D The speedup is unrelated to GPUs and caused by random fluctuations.

💡 Hint

Consider overheads in multi-GPU distributed training.

DistributedDataParallel parallelizes training but requires communication and gradient synchronization, which adds overhead and reduces perfect scaling.

Practice

1. What is the main purpose of using torch.nn.DataParallel in PyTorch?

easy

A. To reduce the model size by pruning layers

B. To convert the model to a CPU-only version

C. To split the model and data across multiple GPUs for faster training

D. To visualize the model architecture

Solution

1. Step 1: Understand the role of DataParallel

   torch.nn.DataParallel is designed to distribute the model and input data across multiple GPUs to speed up training.

2. Step 2: Compare options

   Reducing the model size by pruning layers, converting the model to a CPU-only version, and visualizing the model architecture do not relate to multi-GPU training. Only splitting the model and data across multiple GPUs for faster training correctly describes the purpose.

3. Final Answer:

   To split the model and data across multiple GPUs for faster training -> Option C

4. Quick Check:

   Multi-GPU training = DataParallel usage [OK]

Hint: DataParallel splits model and data across GPUs for speed [OK]

Common Mistakes:

• Thinking DataParallel reduces model size
• Confusing DataParallel with CPU conversion
• Assuming DataParallel is for visualization

2. Which of the following is the correct way to wrap a PyTorch model for multi-GPU training?

easy

A. model = torch.nn.DataParallel(model)

B. model = torch.nn.Sequential(model)

C. model = torch.nn.ModuleList(model)

D. model = torch.nn.Parameter(model)

Solution

1. Step 1: Identify the correct wrapper for multi-GPU

   To enable multi-GPU training, PyTorch provides torch.nn.DataParallel which wraps the model.

2. Step 2: Check other options

   Sequential chains layers, ModuleList stores layers, and Parameter is for tensors, none enable multi-GPU.

3. Final Answer:

   model = torch.nn.DataParallel(model) -> Option A

4. Quick Check:

   Wrap model with DataParallel for multi-GPU [OK]

Hint: Use DataParallel(model) to enable multi-GPU training [OK]

Common Mistakes:

• Using Sequential or ModuleList instead of DataParallel
• Trying to wrap model with Parameter
• Forgetting to wrap the model at all

3. Consider this code snippet for multi-GPU training:

import torch
model = MyModel()
model = torch.nn.DataParallel(model)
model.cuda()
inputs = inputs.cuda()
outputs = model(inputs)
print(outputs.device)

What will be printed if multiple GPUs are available?

medium

A. cuda:0

B. cpu

C. cuda:1

D. cuda:all

Solution

1. Step 1: Understand DataParallel device behavior

   When using DataParallel, the model replicates on all GPUs but the output is gathered on the default device, usually cuda:0.

2. Step 2: Analyze the print statement

   The outputs.device will show the device of the gathered output tensor, which is cuda:0, not all GPUs or CPU.

3. Final Answer:

   cuda:0 -> Option A

4. Quick Check:

   DataParallel output device = cuda:0 [OK]

Hint: DataParallel gathers output on cuda:0 device [OK]

Common Mistakes:

• Expecting output on all GPUs simultaneously
• Thinking output is on CPU
• Assuming output device is cuda:1 or others

4. You wrote this code for multi-GPU training but get an error:

model = MyModel()
model = torch.nn.DataParallel(model)
inputs = inputs.cuda()
outputs = model(inputs)

The error says: 'Expected all tensors to be on the same device'. What is the likely cause?

medium

A. Model should be wrapped with Sequential instead

B. Model is not moved to GPU before wrapping with DataParallel

C. DataParallel requires inputs on CPU

D. Inputs are not on CPU

Solution

1. Step 1: Check model and input device placement

   The model must be moved to GPU (e.g., model.cuda()) before wrapping with DataParallel to ensure all parts are on the same device.

2. Step 2: Analyze error cause

   If the model is on CPU but inputs are on GPU, the error about device mismatch occurs.

3. Final Answer:

   Model is not moved to GPU before wrapping with DataParallel -> Option B

4. Quick Check:

   Model and inputs must be on same device [OK]

Hint: Move model to GPU before DataParallel wrap [OK]

Common Mistakes:

• Wrapping model before moving to GPU
• Assuming inputs must be on CPU
• Using wrong wrapper like Sequential

5. You want to train a model on 4 GPUs using PyTorch. Which of these steps is NOT necessary for correct multi-GPU training?

hard

A. Move input data to GPU before passing to the model

B. Move the model to GPU before wrapping

C. Wrap the model with torch.nn.DataParallel

D. Manually split input batches and send each to a different GPU

Solution

1. Step 1: Review multi-GPU training steps

   Using DataParallel automatically splits input batches and sends them to multiple GPUs, so manual splitting is not needed.

2. Step 2: Confirm necessary steps

   Wrapping the model, moving it to GPU, and moving inputs to GPU are all required for proper multi-GPU training.

3. Final Answer:

   Manually split input batches and send each to a different GPU -> Option D

4. Quick Check:

   DataParallel handles input splitting automatically [OK]

Hint: DataParallel auto-splits inputs; manual split not needed [OK]

Common Mistakes:

• Trying to manually split inputs across GPUs
• Forgetting to move model to GPU
• Not wrapping model with DataParallel