Data parallelism vs model parallelism in MLOps - Practice Questions

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Data Parallelism

Which statement best describes data parallelism in machine learning training?

A Copying the entire model to multiple devices and training on different data batches simultaneously.

B Splitting the model into parts and running each part on different devices.

C Training the model on a single device with one batch of data at a time.

D Using multiple models to train on the same data sequentially.

💡 Hint

Think about whether the model or the data is split across devices.

Data parallelism means copying the full model to multiple devices and each device trains on a different subset of data at the same time.

🧠 Conceptual

intermediate

Understanding Model Parallelism

Which option correctly explains model parallelism in machine learning?

A Running the same model multiple times sequentially on the same data.

B Training multiple copies of the full model on different data batches.

C Using a single device to train the entire model on all data.

D Splitting the model into smaller parts and running each part on different devices simultaneously.

💡 Hint

Consider whether the model or the data is divided across devices.

Model parallelism splits the model itself into parts, each running on different devices to handle large models that don't fit on one device.

💻 Command Output

advanced

Output of Data Parallelism Setup Command

What is the output of this command when setting up data parallelism with PyTorch's DataParallel on 2 GPUs?

MLOps

import torch
import torch.nn as nn
model = nn.Linear(10, 2)
model = nn.DataParallel(model, device_ids=[0,1])
print(model.device_ids)

A [1, 0]

B [0, 1]

C AttributeError: 'DataParallel' object has no attribute 'device_ids'

D None

💡 Hint

Check the attribute that stores device IDs in DataParallel.

The device_ids attribute lists the GPUs used for data parallelism in the order specified.

❓ Troubleshoot

advanced

Troubleshooting Model Parallelism Memory Error

You split a large model across two GPUs using model parallelism, but get a CUDA out-of-memory error on the first GPU. What is the most likely cause?

A The first GPU is assigned too many layers causing memory overflow.

B The batch size is too small to utilize both GPUs.

C Data parallelism was used instead of model parallelism.

D The GPUs are not connected properly.

💡 Hint

Think about how model parts are distributed and GPU memory limits.

If one GPU holds too many model layers, it can run out of memory even if the other GPU is free.

✅ Best Practice

expert

Choosing Between Data and Model Parallelism

Which scenario best justifies using model parallelism over data parallelism?

A When you want to reduce communication overhead by using a single GPU.

B When you want to speed up training by running multiple copies of the model on different data batches.

C When the model is too large to fit into the memory of a single GPU.

D When the dataset is small and fits into one device memory easily.

💡 Hint

Consider the main limitation that model parallelism solves.

Model parallelism is used when the model size exceeds one device's memory, requiring splitting the model across devices.