Multi-GPU training in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of PyTorch DataParallel model forward pass

Consider the following PyTorch code snippet using DataParallel for multi-GPU training. What will be the output shape of output after the forward pass?

PyTorch

import torch
import torch.nn as nn

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(10, 5)

    def forward(self, x):
        return self.linear(x)

model = SimpleModel()
if torch.cuda.device_count() > 1:
    model = nn.DataParallel(model)

model = model.cuda()
input_tensor = torch.randn(16, 10).cuda()
output = model(input_tensor)
print(output.shape)

A torch.Size([8, 5])

B torch.Size([32, 5])

C RuntimeError due to input size mismatch

D torch.Size([16, 5])

💡 Hint

DataParallel splits the batch across GPUs but concatenates outputs back to original batch size.

DataParallel splits the input batch (size 16) across available GPUs, processes them in parallel, then concatenates the outputs. The output shape matches the input batch size with the output feature size (5).

🧠 Conceptual

intermediate

Understanding DistributedDataParallel vs DataParallel

Which statement correctly describes a key difference between PyTorch's DistributedDataParallel (DDP) and DataParallel (DP) for multi-GPU training?

A DP requires manual gradient synchronization, but DDP automatically splits input batches across GPUs.

B DDP runs model replicas in separate processes and synchronizes gradients, while DP runs replicas in a single process splitting input batches.

C DDP only works on a single GPU, while DP supports multiple GPUs on one machine.

D DP uses multiple machines for training, while DDP is limited to one machine.

💡 Hint

Think about process management and how input batches are handled.

DistributedDataParallel runs one process per GPU and synchronizes gradients across processes, which is more efficient and scalable. DataParallel runs in one process and splits input batches across GPUs, which can be slower.

❓ Hyperparameter

advanced

Choosing batch size for multi-GPU training

You want to train a model on 4 GPUs using DistributedDataParallel. Your single-GPU batch size is 32. What is the best practice for setting the batch size when using 4 GPUs?

A Keep batch size 32 and let each GPU process the full batch independently.

B Set batch size to 8 so total batch size is 8 across all GPUs.

C Set batch size to 128 so each GPU processes 32 samples (total batch size = 4 * 32).

D Set batch size to 32 and manually average gradients across GPUs.

💡 Hint

Consider how DistributedDataParallel splits data across GPUs.

In DDP, each GPU processes a subset of the batch. To keep the effective batch size the same as single-GPU training, multiply the batch size by the number of GPUs.

🔧 Debug

advanced

Identifying error in multi-GPU model saving

You trained a model using nn.DataParallel and saved it with torch.save(model.state_dict(), 'model.pth'). When loading the model later without DataParallel, you get a key mismatch error. What is the cause?

A The saved state dict keys are prefixed with 'module.' due to DataParallel wrapping.

B The model was saved on CPU but loaded on GPU causing mismatch.

C The model architecture changed between saving and loading causing mismatch.

D The saved file is corrupted and cannot be loaded properly.

💡 Hint

DataParallel adds a prefix to parameter names in the state dict.

DataParallel wraps the model and prefixes all parameter keys with 'module.'. Loading this state dict into a model without DataParallel causes key mismatch errors.

❓ Metrics

expert

Interpreting training speedup with multi-GPU training

You train a model on 1 GPU and it takes 100 minutes per epoch. Using 4 GPUs with DistributedDataParallel, the epoch time reduces to 30 minutes. Which statement best explains this result?

A The speedup is less than 4x due to communication overhead and synchronization costs.

B The speedup is exactly 4x because 4 GPUs process data independently without overhead.

C The speedup is more than 4x because GPUs work better together than alone.

D The speedup is unrelated to GPUs and caused by random fluctuations.

💡 Hint

Consider overheads in multi-GPU distributed training.

DistributedDataParallel parallelizes training but requires communication and gradient synchronization, which adds overhead and reduces perfect scaling.