DataParallel basics in PyTorch - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to wrap the model for parallel GPU training using DataParallel.

PyTorch

import torch
import torch.nn as nn

model = nn.Linear(10, 2)
model = torch.nn.DataParallel([1])

Drag options to blanks, or click blank then click option'

Atorch.nn.Module()

Bnn.Linear(10, 2)

Ctorch.nn.Linear(10, 2)

Dmodel

Attempts:

3 left

💡 Hint

Common Mistakes

Passing the class instead of the model instance.

Using an unrelated torch module.

We wrap the existing model instance with torch.nn.DataParallel to enable parallel GPU training.

2fill in blank

medium

Complete the code to move the DataParallel model to GPU device 0.

PyTorch

device = torch.device('cuda:0')
model = torch.nn.DataParallel(model)
model = model[1](device)

Drag options to blanks, or click blank then click option'

Acpu

Bcuda

Cto

Ddevice

Attempts:

3 left

💡 Hint

Common Mistakes

Using cuda() without parentheses.

Using cpu() which moves model to CPU.

The to() method moves the model to the specified device.

3fill in blank

hard

Fix the error in the code to correctly get the model's original module after DataParallel wrapping.

PyTorch

if isinstance(model, torch.nn.DataParallel):
    original_model = model[1]
else:
    original_model = model

Drag options to blanks, or click blank then click option'

Adata_parallel

Bmodule

Cmodel

Doriginal

Attempts:

3 left

💡 Hint

Common Mistakes

Trying to access model.model which does not exist.

Using incorrect attribute names.

When using DataParallel, the original model is accessible via the module attribute.

4fill in blank

hard

Fill both blanks to create a DataParallel model and move it to all available GPUs.

PyTorch

model = nn.Linear(20, 5)
model = torch.nn.DataParallel(model, device_ids=[1])
model = model[2]('cuda')

Drag options to blanks, or click blank then click option'

A[0, 1]

B[0]

Cto

Dcuda

Attempts:

3 left

💡 Hint

Common Mistakes

Passing a single integer instead of a list for device_ids.

Using cuda without parentheses as a method.

We specify device IDs as a list of GPU indices and use to('cuda') to move the model to GPUs.

5fill in blank

hard

Fill all three blanks to create a dictionary of losses for each GPU and sum them correctly.

PyTorch

losses = {f'gpu_{i}': output[i].mean() for i in range([1])}
total_loss = sum([3] for [3] in losses[2])

Drag options to blanks, or click blank then click option'

A2

B.values()

Closs

D.keys()

Attempts:

3 left

💡 Hint

Common Mistakes

Iterating over keys instead of values.

Using wrong range for number of GPUs.

We iterate over the values of the losses dictionary for 2 GPUs and sum the loss values.