Multi-GPU training basics in TensorFlow - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a MirroredStrategy for multi-GPU training.

TensorFlow

strategy = tf.distribute.[1]()

Drag options to blanks, or click blank then click option'

AMirroredStrategy

BOneDeviceStrategy

CCentralStorageStrategy

DTPUStrategy

Attempts:

3 left

💡 Hint

Common Mistakes

Using OneDeviceStrategy which only uses one device.

Confusing TPUStrategy with GPU strategies.

MirroredStrategy is used to distribute training across multiple GPUs on one machine.

2fill in blank

medium

Complete the code to build and compile the model inside the strategy scope.

TensorFlow

with strategy.[1]():
    model = tf.keras.Sequential([...])
    model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

Drag options to blanks, or click blank then click option'

Adistribute

Brun

Cstrategy

Dscope

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'run' which is for executing functions, not defining scope.

Using 'strategy' which is the object, not a method.

The 'scope' method is used to specify the context for building and compiling the model under the distribution strategy.

3fill in blank

hard

Fix the error in the code to distribute the dataset for multi-GPU training.

TensorFlow

dist_dataset = strategy.[1](dataset)

Drag options to blanks, or click blank then click option'

Aexperimental_distribute_dataset

Bdistribute_dataset

Cdistribute

Ddistribute_datasets_from_function

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'distribute_dataset' which does not exist.

Using 'distribute' which is not a method.

The correct method to distribute a dataset is 'experimental_distribute_dataset' in TensorFlow 2.x.

4fill in blank

hard

Fill both blanks to correctly reduce the loss across GPUs and update the optimizer.

TensorFlow

per_replica_losses = strategy.run(train_step, args=(dist_inputs,))

loss = strategy.[1](tf.distribute.ReduceOp.SUM, per_replica_losses, axis=None)
optimizer.[2](loss)

Drag options to blanks, or click blank then click option'

Areduce

Bapply_gradients

Creduce_sum

Dminimize

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'reduce_sum' which does not exist.

Using 'apply_gradients' directly without computing gradients.

We use 'reduce' to combine losses from all replicas (with ReduceOp.MEAN or SUM) and 'minimize' to update the optimizer with the loss.

5fill in blank

hard

Fill all three blanks to create a distributed dataset, iterate over it, and run the training step.

TensorFlow

dist_dataset = strategy.[1](dataset)
iterator = iter(dist_dataset)
for _ in range(steps_per_epoch):
    dist_inputs = next(iterator)
    strategy.[2](train_step, args=([3],))

Drag options to blanks, or click blank then click option'

Aexperimental_distribute_dataset

Brun

Cdist_inputs

Ddistribute_dataset

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'distribute_dataset' which is not a valid method.

Confusing 'dist_inputs' with 'dataset'.

We use 'experimental_distribute_dataset' to distribute the dataset, 'run' to execute the training step on each replica, and 'dist_inputs' as the input argument.

Practice

1. What is the main purpose of using tf.distribute.MirroredStrategy() in TensorFlow?

easy

A. To train a model on multiple GPUs simultaneously

B. To reduce the size of the dataset

C. To convert a model to run on CPU only

D. To save the model in a special format

Solution

1. Step 1: Understand the role of MirroredStrategy

   tf.distribute.MirroredStrategy() is designed to distribute training across multiple GPUs to speed up the process.

2. Step 2: Compare options with the purpose

   Options A, B, and C do not relate to multi-GPU training. Only training a model on multiple GPUs simultaneously correctly states the purpose.

3. Final Answer:

   To train a model on multiple GPUs simultaneously -> Option A

4. Quick Check:

   Multi-GPU training = MirroredStrategy usage [OK]

Hint: MirroredStrategy means training on many GPUs at once [OK]

Common Mistakes:

• Thinking it reduces dataset size
• Confusing with CPU-only training
• Assuming it changes model saving format

2. Which of the following is the correct way to start using multi-GPU training in TensorFlow?

easy

A. model = build_model()
strategy = tf.distribute.MirroredStrategy()
model.compile()

B. strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
model = build_model()

C. strategy = tf.distribute.MirroredStrategy()
model = build_model()
model.fit()

D. model = build_model()
model.fit()
strategy = tf.distribute.MirroredStrategy()

Solution

1. Step 1: Identify correct usage of strategy.scope()

   Model building and compiling must happen inside strategy.scope() to enable multi-GPU support.

2. Step 2: Check each option for correct order

   Only strategy = tf.distribute.MirroredStrategy() with strategy.scope(): model = build_model() builds the model inside the strategy scope. Others build or compile outside, which disables multi-GPU benefits.

3. Final Answer:

   strategy = tf.distribute.MirroredStrategy() with strategy.scope(): model = build_model() -> Option B

4. Quick Check:

   Build model inside strategy.scope() = strategy = tf.distribute.MirroredStrategy() with strategy.scope(): model = build_model() [OK]

Hint: Always build and compile inside strategy.scope() [OK]

Common Mistakes:

• Building model before strategy.scope()
• Compiling model outside strategy.scope()
• Calling fit before defining strategy

3. Given the code below, what will be the output shape of the predictions if the input batch size is 64 and there are 2 GPUs?

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    model = tf.keras.Sequential([
        tf.keras.layers.Dense(10, activation='softmax')
    ])
    model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')

import numpy as np
inputs = np.random.random((64, 20))
predictions = model.predict(inputs)
print(predictions.shape)

medium

A. (10, 64)

B. (32, 10)

C. (128, 10)

D. (64, 10)

Solution

1. Step 1: Understand batch size and MirroredStrategy behavior

   MirroredStrategy splits the batch across GPUs internally but the output predictions shape matches the input batch size.

2. Step 2: Check model output shape

   The model outputs 10 classes per input. Input batch size is 64, so output shape is (64, 10).

3. Final Answer:

   (64, 10) -> Option D

4. Quick Check:

   Output shape = (batch_size, classes) = (64, 10) [OK]

Hint: Output shape matches input batch size, not multiplied by GPUs [OK]

Common Mistakes:

• Multiplying batch size by number of GPUs
• Confusing output shape dimensions
• Assuming output shape is (10, 64)

4. You wrote this code but get an error: RuntimeError: tf.distribute.MirroredStrategy() must be used with model creation inside its scope. What is the likely mistake?

strategy = tf.distribute.MirroredStrategy()
model = build_model()
with strategy.scope():
    model.compile(optimizer='adam', loss='mse')

medium

A. Model was compiled outside the strategy.scope()

B. MirroredStrategy cannot be used with compile()

C. Model was built outside the strategy.scope()

D. The optimizer 'adam' is not supported

Solution

1. Step 1: Analyze error message and code

   The error says model creation must be inside strategy.scope(). Here, model is built before entering the scope.

2. Step 2: Identify correct fix

   Building the model inside the scope fixes the error. Compiling inside scope is correct.

3. Final Answer:

   Model was built outside the strategy.scope() -> Option C

4. Quick Check:

   Build model inside scope to avoid RuntimeError [OK]

Hint: Build model inside strategy.scope() to fix this error [OK]

Common Mistakes:

• Building model before strategy.scope()
• Thinking compile() causes error
• Blaming optimizer choice

5. You want to train a large model on 4 GPUs using TensorFlow. Which of the following steps correctly applies multi-GPU training and ensures the model can handle larger batch sizes efficiently?

hard

A. Use tf.distribute.MirroredStrategy(), build and compile the model inside its scope, then increase batch size in model.fit()

B. Build the model first, then use tf.distribute.MirroredStrategy() only during model.fit() with default batch size

C. Use tf.distribute.MirroredStrategy() but keep batch size small to avoid memory issues

D. Train the model separately on each GPU without using MirroredStrategy and average weights manually

Solution

1. Step 1: Apply MirroredStrategy correctly

   Use tf.distribute.MirroredStrategy() and build/compile the model inside its scope to enable multi-GPU training.

2. Step 2: Adjust batch size for efficiency

   Increasing batch size in model.fit() leverages multiple GPUs to handle larger data efficiently.

3. Final Answer:

   Use tf.distribute.MirroredStrategy(), build and compile the model inside its scope, then increase batch size in model.fit() -> Option A

4. Quick Check:

   Build inside scope + increase batch size = efficient multi-GPU training [OK]

Hint: Build inside scope and increase batch size for multi-GPU speed [OK]

Common Mistakes:

• Building model outside strategy scope
• Not increasing batch size to use GPUs fully
• Manually averaging weights instead of using MirroredStrategy