Why distributed training handles large models in PyTorch - Experiment to Prove It

Experiment - Why distributed training handles large models

Problem:Training a large neural network model on a single GPU causes out-of-memory errors and slow training.

Current Metrics:Training stops early due to CUDA out-of-memory error; no meaningful accuracy achieved.

Issue:The model is too large to fit into the memory of a single GPU, causing training to fail.

Your Task

Enable training of the large model by using distributed training across two GPUs without changing the model architecture.

Do not reduce the model size or complexity.

Use PyTorch's DistributedDataParallel for training.

Keep batch size per GPU the same as before.

Hint 1

Hint 2

Hint 3

Hint 4

Solution

PyTorch

import torch
import torch.nn as nn
import torch.optim as optim
import torch.distributed as dist
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP

class LargeModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(10000, 5000)
        self.relu = nn.ReLU()
        self.layer2 = nn.Linear(5000, 1000)
        self.layer3 = nn.Linear(1000, 10)

    def forward(self, x):
        x = self.relu(self.layer1(x))
        x = self.relu(self.layer2(x))
        return self.layer3(x)

def setup(rank, world_size):
    dist.init_process_group(
        backend='nccl',
        init_method='tcp://127.0.0.1:29500',
        rank=rank,
        world_size=world_size
    )

def cleanup():
    dist.destroy_process_group()

def train(rank, world_size):
    setup(rank, world_size)

    torch.manual_seed(42 + rank)
    device = torch.device(f'cuda:{rank}')

    model = LargeModel().to(device)
    ddp_model = DDP(model, device_ids=[rank])

    optimizer = optim.SGD(ddp_model.parameters(), lr=0.01)
    loss_fn = nn.CrossEntropyLoss()

    # Dummy dataset: 50 samples per GPU (100 total), input size 10000
    num_samples_per_gpu = 50
    inputs = torch.randn(num_samples_per_gpu, 10000).to(device)
    targets = torch.randint(0, 10, (num_samples_per_gpu,)).to(device)

    batch_size = 10
    for epoch in range(3):
        ddp_model.train()
        for i in range(0, num_samples_per_gpu, batch_size):
            optimizer.zero_grad()
            batch_inputs = inputs[i:i+batch_size]
            batch_targets = targets[i:i+batch_size]
            outputs = ddp_model(batch_inputs)
            loss = loss_fn(outputs, batch_targets)
            loss.backward()
            optimizer.step()
        if rank == 0:
            print(f'Epoch {epoch+1} complete on rank {rank}')

    cleanup()

if __name__ == '__main__':
    world_size = 2
    mp.spawn(train, args=(world_size,), nprocs=world_size, join=True)

Added distributed training setup using torch.distributed and torch.multiprocessing.

Wrapped the large model with DistributedDataParallel to split training across GPUs.

Initialized process group for communication between GPUs.

Used multiple processes to run training on two GPUs simultaneously.

Results Interpretation

Before: Training failed due to GPU memory overflow; no accuracy or loss metrics available.

After: Training completes 3 epochs without errors; loss decreases indicating learning; model replicated across two GPUs using data parallelism.

Distributed data parallel training replicates the model across multiple GPUs and splits the data across them, allowing large models to train efficiently by overcoming single GPU compute and batch size limits.

Bonus Experiment

Try training the same large model using model parallelism instead of data parallelism.

💡 Hint

Split the model layers across different GPUs manually and pass data through them sequentially.

Practice

1. Why does distributed training help when working with very large models in PyTorch?

easy

A. It converts the model to a simpler version with fewer layers.

B. It reduces the number of model parameters automatically.

C. It splits the batch across multiple devices to fit memory limits.

D. It trains the model on a single device but faster.

Solution

1. Step 1: Understand memory limits on single devices

   Large models often cannot fit into the memory of one device like a GPU.

2. Step 2: How distributed training solves this

   Distributed training splits the batch across multiple devices, so each device processes part of the batch, fitting memory limits.

3. Final Answer:

   It splits the batch across multiple devices to fit memory limits. -> Option C

4. Quick Check:

   Distributed training = splitting batch across devices [OK]

Hint: Distributed training splits batch to fit device memory [OK]

Common Mistakes:

• Thinking distributed training reduces model size
• Believing it simplifies the model layers
• Assuming training happens on one device only

2. Which PyTorch class is commonly used for distributed training of large models?

easy

A. torch.nn.DistributedDataParallel

B. torch.nn.DataParallel

C. torch.optim.SGD

D. torch.utils.data.DataLoader

Solution

1. Step 1: Identify PyTorch classes related to parallelism

   torch.nn.DataParallel is for single-machine multi-GPU but less efficient for large scale.

2. Step 2: Recognize the class designed for distributed training

   torch.nn.DistributedDataParallel is designed for multi-machine and multi-device distributed training, handling large models efficiently.

3. Final Answer:

   torch.nn.DistributedDataParallel -> Option A

4. Quick Check:

   DistributedDataParallel = distributed training tool [OK]

Hint: DistributedDataParallel is for distributed training [OK]

Common Mistakes:

• Confusing DataParallel with DistributedDataParallel
• Choosing optimizer or data loader classes
• Not knowing PyTorch parallel classes

3. Given this PyTorch code snippet for distributed training, what will be the output shape of the model's output tensor if the input batch size is 32?

import torch
import torch.nn as nn
import torch.distributed as dist

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(10, 5)
    def forward(self, x):
        return self.linear(x)

model = SimpleModel()
dist.init_process_group(backend='gloo', world_size=1, rank=0)
model = nn.parallel.DistributedDataParallel(model)
input_tensor = torch.randn(32, 10)
output = model(input_tensor)
print(output.shape)

medium

A. torch.Size([32, 5])

B. torch.Size([10, 5])

C. torch.Size([5, 32])

D. torch.Size([32, 10])

Solution

1. Step 1: Understand model input and output shapes

   The input tensor has shape (32, 10), batch size 32 and feature size 10. The linear layer maps 10 features to 5 outputs.

2. Step 2: Effect of DistributedDataParallel on output shape

   DistributedDataParallel wraps the model but does not change output shape. So output shape matches linear layer output: (32, 5).

3. Final Answer:

   torch.Size([32, 5]) -> Option A

4. Quick Check:

   Output shape = (batch size, output features) = (32, 5) [OK]

Hint: DistributedDataParallel does not change output shape [OK]

Common Mistakes:

• Confusing input and output dimensions
• Thinking distributed wrapper changes output shape
• Mixing batch size and feature dimensions

4. You wrote this code to use DistributedDataParallel but get a runtime error:

model = SimpleModel()
model = nn.parallel.DistributedDataParallel(model)
output = model(input_tensor)

What is the most likely cause of the error?

medium

A. You used DataParallel instead of DistributedDataParallel.

B. You forgot to initialize the distributed process group before wrapping the model.

C. The model class does not inherit from nn.Module.

D. The input tensor shape is incorrect.

Solution

1. Step 1: Check requirements for DistributedDataParallel

   DistributedDataParallel requires initializing the distributed process group before wrapping the model.

2. Step 2: Identify common runtime error cause

   If the process group is not initialized, runtime errors occur when calling DistributedDataParallel.

3. Final Answer:

   You forgot to initialize the distributed process group before wrapping the model. -> Option B

4. Quick Check:

   Initialize process group before DistributedDataParallel [OK]

Hint: Always init process group before DistributedDataParallel [OK]

Common Mistakes:

• Ignoring process group initialization
• Blaming input tensor shape
• Confusing DataParallel with DistributedDataParallel

5. You want to train a very large model that does not fit into a single GPU's memory. Which approach best uses PyTorch's distributed training to handle this?

hard

A. Reduce the batch size until the model fits on one GPU.

B. Use DataParallel to copy the entire model on each GPU and split the input batch.

C. Train the model on CPU to avoid GPU memory limits.

D. Use DistributedDataParallel to split the batch across multiple GPUs and synchronize gradients.

Solution

1. Step 1: Understand how distributed training scales large models

   Very large models benefit from distributed training across multiple GPUs. DistributedDataParallel replicates the model on each GPU, splits the batch across them, and synchronizes gradients.

2. Step 2: Compare approaches

   DataParallel copies full model on each GPU, which is less efficient for large scale distributed setups. Reducing batch size or using CPU does not scale efficiently.

3. Final Answer:

   Use DistributedDataParallel to split the batch across multiple GPUs and synchronize gradients. -> Option D

4. Quick Check:

   DistributedDataParallel splits batch for large models [OK]

Hint: DistributedDataParallel splits batch, DataParallel single-machine [OK]

Common Mistakes:

• Choosing DataParallel for large models
• Only reducing batch size without model splitting
• Ignoring GPU memory limits