DistributedDataParallel in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why use DistributedDataParallel (DDP)?

Imagine you want to train a deep learning model faster by using multiple GPUs. Why is DistributedDataParallel (DDP) preferred over DataParallel in PyTorch?

A DDP reduces GPU memory usage by splitting the model across GPUs, while DataParallel duplicates the model on each GPU.

B DDP synchronizes gradients efficiently across GPUs and avoids Python GIL bottlenecks, while DataParallel runs on a single process causing slower training.

C DDP automatically increases batch size without code changes, while DataParallel requires manual batch size tuning.

D DDP only works on CPU clusters, while DataParallel is designed for GPUs.

💡 Hint

Think about how Python's Global Interpreter Lock (GIL) affects multi-threading and how DDP uses multiple processes.

DDP uses multiple processes, each handling one GPU, which avoids Python's GIL and synchronizes gradients efficiently. DataParallel uses threads within one process, causing slower training and more overhead.

❓ Predict Output

intermediate

Output of DDP model training snippet

What will be the printed output of the following PyTorch code snippet using DistributedDataParallel?

PyTorch

import torch
import torch.nn as nn
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

def setup(rank, world_size):
    dist.init_process_group('gloo', rank=rank, world_size=world_size)

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(2, 2)
    def forward(self, x):
        return self.linear(x)

rank = 0
world_size = 1
setup(rank, world_size)
model = SimpleModel().to(rank)
ddp_model = DDP(model, device_ids=None)
input_tensor = torch.tensor([[1.0, 2.0]], device=rank)
output = ddp_model(input_tensor)
print(output)

A tensor([[some_value, some_value]], grad_fn=<AddmmBackward0>)

B tensor([[1.0, 2.0]], grad_fn=<AddmmBackward0>)

C RuntimeError: Expected all tensors to be on the same device

D tensor([[0.0, 0.0]], grad_fn=<AddmmBackward0>)

💡 Hint

The model has a linear layer with random weights initialized. The output will be a tensor with computed values, not the input itself.

The linear layer applies a matrix multiplication plus bias to the input tensor, producing a new tensor with different values. Since weights are randomly initialized, the exact values vary but the output shape and type are as shown.

❓ Hyperparameter

advanced

Choosing the correct batch size with DistributedDataParallel

You have 4 GPUs and want to train a model using DistributedDataParallel. Your original batch size for single GPU training is 64. What should be the batch size per GPU when using DDP to keep the effective batch size the same?

A 64 per GPU, total batch size 256

B 256 per GPU, total batch size 1024

C 16 per GPU, total batch size 64

D 1 per GPU, total batch size 4

💡 Hint

Think about how DDP splits data across GPUs and how total batch size is calculated.

In DDP, each GPU processes its own batch. To keep the total batch size same as single GPU training, divide the original batch size by number of GPUs. So 64 / 4 = 16 per GPU.

🔧 Debug

advanced

Why does this DDP code raise RuntimeError?

Consider this PyTorch DDP training snippet. It raises a RuntimeError: "Expected to have same number of elements in all input tensors" during backward. What is the likely cause?

PyTorch

import torch
import torch.nn as nn
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

def setup(rank, world_size):
    dist.init_process_group('gloo', rank=rank, world_size=world_size)

rank = 0
world_size = 2
setup(rank, world_size)
model = nn.Linear(2, 2).to(rank)
ddp_model = DDP(model, device_ids=None)

input_tensor = torch.randn(3, 2).to(rank) if rank == 0 else torch.randn(4, 2).to(rank)
output = ddp_model(input_tensor)
loss = output.sum()
loss.backward()

A Loss function is not defined properly causing backward to fail.

B Model weights are not moved to the correct device before wrapping with DDP.

C Distributed process group is not initialized before model creation.

D Input tensors have different batch sizes on each GPU causing mismatch during gradient synchronization.

💡 Hint

Check the input tensor sizes on each rank and how DDP expects inputs.

DDP requires all processes to have the same batch size because it synchronizes gradients across all GPUs. Different input sizes cause mismatch errors during backward.

❓ Model Choice

expert

Best model wrapping strategy for mixed CPU-GPU training with DDP

You want to train a model where some layers run on CPU and others on GPU, but still use DistributedDataParallel for multi-GPU training. Which approach is correct?

A Wrap only the GPU layers with DDP and keep CPU layers outside the DDP wrapper, combining outputs manually.

B Split the model into two parts: wrap GPU layers with DDP on GPUs, and run CPU layers separately without DDP.

C Wrap the entire model with DDP and move only GPU layers to GPU devices; CPU layers stay on CPU.

D Use DataParallel instead of DDP because DDP requires the whole model on GPU.

💡 Hint

Consider how DDP expects model parameters and devices.

DDP requires all parameters it manages to be on the same device per process. Wrapping only GPU layers with DDP and handling CPU layers separately allows mixed device usage while still benefiting from DDP on GPUs.

Practice

1. What is the main purpose of using DistributedDataParallel in PyTorch?

easy

A. To automatically tune hyperparameters during training

B. To reduce the size of the model for deployment

C. To convert a model to run only on CPU

D. To speed up training by using multiple GPUs or machines simultaneously

Solution

1. Step 1: Understand the role of DistributedDataParallel

   DistributedDataParallel (DDP) is designed to run training across multiple GPUs or machines to speed up the process.

2. Step 2: Compare options with DDP's purpose

   Options B, C, and D do not relate to parallel training or speed-up. Only To speed up training by using multiple GPUs or machines simultaneously correctly describes DDP's main purpose.

3. Final Answer:

   To speed up training by using multiple GPUs or machines simultaneously -> Option D

4. Quick Check:

   DDP speeds training = A [OK]

Hint: DDP means training faster with many GPUs [OK]

Common Mistakes:

• Thinking DDP reduces model size
• Confusing DDP with CPU-only training
• Assuming DDP tunes hyperparameters automatically

2. Which of the following is the correct way to wrap a PyTorch model with DistributedDataParallel assuming model is your neural network and device is the GPU device?

easy

A. model = torch.nn.DataParallel(model, device_ids=[device])

B. model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[device])

C. model = torch.nn.DistributedDataParallel(model, device=device)

D. model = torch.nn.parallel.DistributedDataParallel(model, devices=device)

Solution

1. Step 1: Recall the correct syntax for DistributedDataParallel

   The correct class is torch.nn.parallel.DistributedDataParallel and it takes device_ids as a list of devices.

2. Step 2: Check each option for syntax correctness

   model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[device]) uses the correct class and parameter device_ids=[device]. model = torch.nn.DataParallel(model, device_ids=[device]) uses DataParallel, which is different. Options C and D use incorrect parameter names or class names.

3. Final Answer:

   model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[device]) -> Option B

4. Quick Check:

   Correct class and device_ids list = A [OK]

Hint: Use torch.nn.parallel.DistributedDataParallel with device_ids list [OK]

Common Mistakes:

• Using DataParallel instead of DistributedDataParallel
• Passing device instead of device_ids list
• Wrong class or parameter names

3. Consider the following code snippet for a distributed training setup:

import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP

def setup(rank, world_size):
    dist.init_process_group('gloo', rank=rank, world_size=world_size)

model = torch.nn.Linear(10, 1).to(rank)
model = DDP(model, device_ids=[rank])

print(model.module.weight.device)

What will be the output device of model.module.weight.device if rank=1 and GPUs are available?

medium

A. cuda:1

B. cpu

C. cuda:0

D. cuda:rank

Solution

1. Step 1: Understand device assignment in the code

   The model is moved to rank device with .to(rank), so for rank=1, it is on cuda:1.

2. Step 2: Check DistributedDataParallel wrapping

   DDP wraps the model and uses device_ids=[rank], so the model stays on cuda:1.

3. Final Answer:

   cuda:1 -> Option A

4. Quick Check:

   Model device matches rank GPU = B [OK]

Hint: Model device matches rank GPU in DDP setup [OK]

Common Mistakes:

• Assuming model stays on CPU
• Confusing device index with rank variable
• Thinking device is always cuda:0

4. You run a distributed training script using DistributedDataParallel but get the error: RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. What is the most likely cause?

medium

A. You forgot to call optimizer.step() after loss.backward()

B. You did not wrap the model with DistributedDataParallel

C. You used the wrong backend in init_process_group

D. You did not call dist.destroy_process_group() at the end

Solution

1. Step 1: Understand the error about unfinished reduction

   This error usually means gradients were not synchronized properly before the next iteration.

2. Step 2: Identify missing optimizer step

   Not calling optimizer.step() after loss.backward() can cause gradients to not be updated and reduction to remain unfinished.

3. Final Answer:

   You forgot to call optimizer.step() after loss.backward() -> Option A

4. Quick Check:

   Missing optimizer.step() causes reduction error = C [OK]

Hint: Always call optimizer.step() after backward() in DDP [OK]

Common Mistakes:

• Skipping optimizer.step() call
• Ignoring proper model wrapping
• Confusing backend errors with sync errors

5. You want to train a model on 4 GPUs using DistributedDataParallel across 2 machines, each with 2 GPUs. Which of the following is the correct approach to set up the training?

hard

A. Launch 4 processes on one machine only, ignoring the second machine

B. Launch 2 processes total, each process uses 2 GPUs, wrap model with DataParallel on each process

C. Launch 4 processes total, each process uses one GPU, initialize process group with correct rank and world_size, wrap model with DDP on each process

D. Launch 1 process per machine, use DDP with device_ids=[0,1] on each machine

Solution

1. Step 1: Understand multi-machine multi-GPU DDP setup

   Each GPU should have one process. For 4 GPUs across 2 machines, launch 4 processes total.

2. Step 2: Confirm process group and model wrapping

   Initialize the process group with correct rank and world_size=4. Wrap the model with DDP on each process using the assigned GPU.

3. Final Answer:

   Launch 4 processes total, each process uses one GPU, initialize process group with correct rank and world_size, wrap model with DDP on each process -> Option C

4. Quick Check:

   One process per GPU with correct init = D [OK]

Hint: One process per GPU, init process group correctly [OK]

Common Mistakes:

• Using fewer processes than GPUs
• Using DataParallel instead of DDP for multi-machine
• Ignoring second machine in setup