Multi-GPU training in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is Multi-GPU training in PyTorch?

Multi-GPU training means using more than one graphics card to train a model faster by splitting the work across GPUs.

beginner

What PyTorch class helps to easily use multiple GPUs for training?

The <code>torch.nn.DataParallel</code> class wraps a model to run it on multiple GPUs by splitting input data automatically.

beginner

Why do we need to move the model and data to GPUs in PyTorch multi-GPU training?

Because GPUs do the heavy math work, the model and data must be on GPUs to speed up training. PyTorch needs explicit commands to move them.

intermediate

What is a common challenge when using multiple GPUs for training?

Synchronizing gradients and combining results from all GPUs can be tricky, but PyTorch handles this automatically with DataParallel.

intermediate

How does DataParallel split the input data across GPUs?

It splits the input batch into smaller chunks, sends each chunk to a different GPU, runs the model on each chunk, then combines the outputs.

Which PyTorch class is commonly used for simple multi-GPU training?

A torch.optim.Adam

B torch.nn.DataParallel

C torch.utils.data.DataLoader

D torch.nn.Sequential

torch.nn.DataParallel is designed to wrap models for multi-GPU training.

Before training on GPUs, what must you do with your model in PyTorch?

A Call <code>model.to('cuda')</code> to move it to GPU

B Call <code>model.train()</code> only

C Call <code>model.eval()</code>

D Nothing, PyTorch moves it automatically

You must move the model to GPU explicitly using model.to('cuda').

How does DataParallel handle the outputs from multiple GPUs?

A It discards outputs from all but one GPU

B It leaves outputs on each GPU separately

C It combines outputs back on the main GPU

D It sends outputs to CPU automatically

DataParallel gathers outputs from all GPUs and combines them on the main GPU.

What is a benefit of using multiple GPUs for training?

A Less memory usage on each GPU

B No need to move data to GPU

C Simpler code without changes

D Faster training by parallelizing work

Multiple GPUs speed up training by sharing the workload.

Which of these is NOT true about DataParallel?

A It requires manual gradient synchronization

B It wraps the model for multi-GPU use

C It combines outputs on the main GPU

D It automatically splits input batches

DataParallel automatically synchronizes gradients; manual sync is not needed.

Explain how PyTorch's DataParallel helps in multi-GPU training.

Think about how input and output are handled across GPUs.

Describe the steps to prepare a PyTorch model and data for multi-GPU training.

Consider what needs to be on GPU and how to enable multi-GPU.