DataParallel basics in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is DataParallel in PyTorch?

DataParallel is a PyTorch feature that helps run a model on multiple GPUs by splitting input data across them and combining results automatically.

beginner

How does DataParallel split the input data?

DataParallel splits the input batch into smaller chunks, sending each chunk to a different GPU to process in parallel.

beginner

What happens to the model outputs in DataParallel?

After each GPU processes its chunk, DataParallel gathers all outputs and combines them into a single output on the main GPU.

beginner

How do you wrap a PyTorch model with DataParallel?

You wrap your model by calling model = torch.nn.DataParallel(model). This enables multi-GPU usage automatically.

intermediate

What is a key limitation of DataParallel?

DataParallel uses one main GPU to gather results, which can create a bottleneck and limit scaling to many GPUs.

What does DataParallel do with input data in PyTorch?

A Runs model on CPU only

B Combines multiple models into one

C Converts data to a different format

D Splits input data across multiple GPUs

DataParallel splits the input batch across GPUs to run the model in parallel.

How do you enable DataParallel for a model in PyTorch?

A model = DataParallel(model)

B model = torch.DataParallel(model)

C model = torch.nn.DataParallel(model)

D model.parallelize()

The correct syntax is torch.nn.DataParallel(model) to wrap the model.

Where does DataParallel gather the outputs from multiple GPUs?

A On the main GPU

B On the CPU

C On each GPU separately

D It does not gather outputs

DataParallel collects outputs on the main GPU to combine them.

What is a downside of using DataParallel?

A Main GPU can become a bottleneck

B It only works on CPUs

C It requires manual data splitting

D It does not support GPUs

The main GPU gathers outputs, which can slow down scaling.

Which PyTorch module provides DataParallel?

A torch.optim

B torch.nn

C torch.utils.data

D torch.cuda

DataParallel is part of torch.nn module.

Explain how DataParallel works in PyTorch to use multiple GPUs.

Think about how data and results move between GPUs.

Describe one advantage and one limitation of using DataParallel.

Consider what makes it simple and what slows it down.