DataParallel basics in PyTorch - Model Pipeline Trace

Model Pipeline - DataParallel basics

This pipeline shows how PyTorch's DataParallel helps train a model faster by splitting data across multiple GPUs. It copies the model to each GPU, processes parts of the data in parallel, then combines the results.

Data Flow - 5 Stages

1Input Data

1000 rows x 10 features→Original dataset loaded for training→1000 rows x 10 features

[[0.5, 1.2, ..., 0.3], [0.7, 0.8, ..., 0.1], ...]

↓

2Batch Split for GPUs

100 rows x 10 features (batch size)→Batch split evenly across 2 GPUs→2 batches of 50 rows x 10 features each

[GPU0 batch: [[0.5, 1.2, ...], ... 50 rows], GPU1 batch: [[0.7, 0.8, ...], ... 50 rows]]

↓

3Model Replication

Model with parameters (weights)→Model copied to each GPU→2 model replicas on GPU0 and GPU1

Same model architecture on both GPUs

↓

4Parallel Forward Pass

2 batches of 50 rows x 10 features→Each GPU processes its batch through its model copy→2 outputs of 50 rows x 1 prediction each

[GPU0 output: [0.8, 0.3, ...], GPU1 output: [0.5, 0.9, ...]]

↓

5Gather Outputs

2 outputs of 50 rows x 1 prediction→Combine outputs from GPUs into one batch→100 rows x 1 prediction

[0.8, 0.3, ..., 0.5, 0.9, ...]

Training Trace - Epoch by Epoch

Loss
0.7 |****
0.6 |*** 
0.5 |**  
0.4 |**  
0.3 |*   
0.2 |*   
    +---------
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.65	0.60	Initial training with DataParallel, loss starts high
2	0.48	0.72	Loss decreases as model learns, accuracy improves
3	0.35	0.81	Continued improvement, model converging
4	0.28	0.86	Loss lowers further, accuracy nearing good performance
5	0.22	0.90	Training stabilizes with good accuracy

Prediction Trace - 4 Layers

Layer 1: Input batch split

Layer 2: Forward pass on GPU0

Layer 3: Forward pass on GPU1

Layer 4: Gather outputs

Model Quiz - 3 Questions

What is the main benefit of using DataParallel in PyTorch?

A It reduces the size of the model

B It increases the number of model parameters

C It splits data across GPUs to speed up training

D It changes the model architecture automatically

Key Insight

Using DataParallel allows the same model to run on multiple GPUs simultaneously by splitting input data batches. This speeds up training without changing the model itself. Loss decreases and accuracy improves as training progresses, showing effective learning.