DistributedDataParallel in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is DistributedDataParallel in PyTorch?

DistributedDataParallel (DDP) is a PyTorch module that helps train models on multiple GPUs or machines by splitting data and synchronizing gradients efficiently.

intermediate

Why use DistributedDataParallel instead of DataParallel?

DistributedDataParallel is faster and more scalable because it uses multiple processes and synchronizes gradients in parallel, while DataParallel uses a single process and can be slower.

intermediate

How does DistributedDataParallel synchronize model updates?

DDP synchronizes gradients by averaging them across all processes after each backward pass, ensuring all model copies stay consistent during training.

beginner

What is the role of torch.distributed.init_process_group() in DDP?

It initializes the communication between processes, setting up the environment so that DistributedDataParallel can coordinate training across GPUs or machines.

beginner

What must you do with your dataset when using DistributedDataParallel?

You should use a DistributedSampler to split the dataset so each process gets a unique subset, avoiding overlap and ensuring efficient training.

What does DistributedDataParallel primarily help with?

A Visualizing model predictions

B Reducing model size

C Training models on multiple GPUs or machines efficiently

D Improving data loading speed

DistributedDataParallel is designed to train models efficiently across multiple GPUs or machines by distributing data and synchronizing gradients.

Which function initializes communication between processes in DDP?

A torch.distributed.init_process_group()

B torch.nn.Module()

C torch.optim.SGD()

D torch.utils.data.DataLoader()

torch.distributed.init_process_group() sets up the communication needed for DistributedDataParallel to work.

What is the purpose of DistributedSampler in DDP?

A To shuffle data randomly

B To split dataset uniquely across processes

C To increase batch size

D To reduce model parameters

DistributedSampler ensures each process gets a unique subset of data, avoiding overlap during distributed training.

How does DDP synchronize model updates?

A By copying weights from one GPU to another

B By saving checkpoints

C By sending data to CPU

D By averaging gradients across all processes after backward pass

DDP averages gradients from all processes to keep model copies consistent.

Compared to DataParallel, DistributedDataParallel is:

A Faster and more scalable

B Only for CPU training

C Slower and less scalable

D Used for model evaluation only

DistributedDataParallel uses multiple processes and parallel communication, making it faster and more scalable than DataParallel.

Explain how DistributedDataParallel works to train a model on multiple GPUs.

Think about how data and model updates are shared across GPUs.

Describe the steps needed to prepare your PyTorch training code to use DistributedDataParallel.

Focus on setup and data handling for distributed training.