Mixed precision training (AMP) in PyTorch - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is mixed precision training in deep learning?

Mixed precision training uses both 16-bit and 32-bit floating point numbers during model training to speed up computation and reduce memory use, while keeping model accuracy.

beginner

What does AMP stand for in PyTorch?

AMP stands for Automatic Mixed Precision. It is a PyTorch feature that automatically manages mixed precision training to make it easier and safer.

beginner

Why use mixed precision training? List two benefits.

1. Faster training because 16-bit operations are quicker on modern GPUs.<br>2. Less GPU memory used, allowing bigger models or larger batches.

intermediate

How does PyTorch AMP help prevent underflow or overflow during training?

PyTorch AMP uses a technique called loss scaling. It multiplies the loss by a big number before backpropagation to keep gradients in a good range, then scales them back down.

intermediate

Show a simple PyTorch code snippet to enable AMP during training.

scaler = torch.cuda.amp.GradScaler() for data, target in dataloader: optimizer.zero_grad() with torch.cuda.amp.autocast(): output = model(data) loss = loss_fn(output, target) scaler.scale(loss).backward() scaler.step(optimizer) scaler.update()

What is the main purpose of using mixed precision training?

A Make training more accurate by using 64-bit floats

B Increase model size without changing speed

C Speed up training and reduce memory use

D Avoid using GPUs

Mixed precision training speeds up training and reduces memory by using 16-bit floats where possible.

In PyTorch AMP, what does the GradScaler do?

A Scales the model weights

B Scales the loss to prevent gradient underflow

C Scales the input data

D Scales the learning rate

GradScaler multiplies the loss to keep gradients in a good numeric range during backpropagation.

Which PyTorch context manager is used to enable mixed precision operations?

A torch.cuda.amp.autocast()

B torch.cuda.amp.scale()

C torch.enable_grad()

D torch.no_grad()

torch.cuda.amp.autocast() enables automatic mixed precision for operations inside its block.

What data types are mainly used in mixed precision training?

A float16 and float32

B float64 and float128

C int32 and int64

D int8 and float8

Mixed precision training uses float16 (half precision) and float32 (single precision) floats.

What happens if you do not use loss scaling in mixed precision training?

A Training will be faster

B Memory usage will increase

C Model accuracy will always improve

D Gradients might become zero due to underflow

Without loss scaling, small gradient values can underflow to zero in float16, harming training.

Explain how mixed precision training works and why it is useful.

Think about how using smaller numbers can help speed and memory, but also what problem it causes.

Describe the steps to enable AMP in a PyTorch training loop.

Focus on the order of operations inside the training loop.