Dynamic computation graph advantage in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why use dynamic computation graphs in PyTorch?

Which of the following best explains the main advantage of dynamic computation graphs in PyTorch compared to static graphs?

A They allow changing the model structure during runtime, enabling flexible architectures like variable-length inputs.

B They automatically optimize the graph for faster execution without user intervention.

C They require less memory because the graph is precompiled once and reused for all inputs.

D They prevent any runtime errors by checking all operations before training starts.

💡 Hint

Think about how dynamic graphs let you build the model step-by-step as data flows through.

Dynamic computation graphs are built on-the-fly during each forward pass, allowing the model to change structure based on input data. This flexibility is useful for tasks like processing sequences of different lengths.

❓ Predict Output

intermediate

Output of dynamic graph example in PyTorch

What will be the output of the following PyTorch code snippet?

PyTorch

import torch
x = torch.tensor([1.0, 2.0, 3.0], requires_grad=True)
y = x * 2 if x.sum() > 5 else x + 2
print(y)

A tensor([1., 2., 3.], requires_grad=True)

B tensor([3., 4., 5.], grad_fn=<AddBackward0>)

C RuntimeError due to conditional operation on tensor

D tensor([2., 4., 6.], grad_fn=<MulBackward0>)

💡 Hint

Check the sum of x and which branch of the if-else runs.

The sum of x is 6, which is greater than 5, so y = x * 2. The output tensor has gradient tracking because x requires grad.

❓ Model Choice

advanced

Choosing model type for variable-length input sequences

You want to build a model that processes sentences of different lengths without padding. Which model type benefits most from dynamic computation graphs?

A Feedforward Neural Network with fixed input dimension

B Convolutional Neural Network (CNN) with fixed input size

C Recurrent Neural Network (RNN) implemented in PyTorch

D Support Vector Machine (SVM) with fixed feature vectors

💡 Hint

Think about models that naturally handle sequences of varying lengths.

RNNs process inputs step-by-step and can handle variable-length sequences naturally. Dynamic graphs in PyTorch allow building the graph differently for each sequence length.

❓ Hyperparameter

advanced

Effect of dynamic graph on training speed

How does using a dynamic computation graph in PyTorch typically affect training speed compared to static graphs?

A Training is always faster because dynamic graphs optimize execution automatically.

B Training is usually slower because the graph is rebuilt every iteration.

C Training speed is unaffected because graphs are cached after first build.

D Training speed depends only on hardware, not graph type.

💡 Hint

Consider the overhead of building the graph each time.

Dynamic graphs are rebuilt every forward pass, which adds overhead and can slow training compared to static graphs that are compiled once.

🔧 Debug

expert

Identifying error in dynamic graph usage

What error will this PyTorch code raise when run?

import torch
x = torch.tensor([1.0, 2.0], requires_grad=True)
with torch.no_grad():
    y = x * 2
z = y.sum()
z.backward()

A RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

B No error, backward runs successfully

C TypeError: unsupported operand type(s) for *: 'Tensor' and 'int'

D ValueError: expected scalar for backward()

💡 Hint

Think about what torch.no_grad() does to the computation graph.

Inside torch.no_grad(), operations do not track gradients. So y does not require grad, causing backward() to fail with RuntimeError.

Practice

1. What is the main advantage of a dynamic computation graph in PyTorch?

easy

A. It allows the model to change its structure during runtime based on input data.

B. It requires pre-defining the entire model graph before training.

C. It makes the model run faster by using static graphs.

D. It prevents any changes to the model once training starts.

Solution

1. Step 1: Understand dynamic graph behavior

   Dynamic graphs build the model step-by-step as data flows, allowing changes during runtime.

2. Step 2: Compare with static graphs

   Static graphs require the entire model to be fixed before running, unlike dynamic graphs.

3. Final Answer:

   It allows the model to change its structure during runtime based on input data. -> Option A

4. Quick Check:

   Dynamic graph = flexible model structure [OK]

Hint: Dynamic graphs let models adapt during runtime [OK]

Common Mistakes:

• Confusing dynamic graphs with static graphs
• Thinking model structure is fixed before training
• Assuming dynamic graphs always run faster

2. Which PyTorch code snippet correctly creates a dynamic computation graph with gradient tracking for a simple addition?

easy

A. x = torch.tensor(2); y = torch.tensor(3); z = x + y

B. x = torch.tensor(2, graph_static=True); y = torch.tensor(3, graph_static=True); z = x + y

C. x = torch.tensor(2); y = torch.tensor(3); z = x * y

D. x = torch.tensor(2, requires_grad=True); y = torch.tensor(3, requires_grad=True); z = x + y

Solution

1. Step 1: Identify dynamic graph creation

   In PyTorch, setting requires_grad=True enables tracking operations for dynamic graph.

2. Step 2: Check code correctness

   x = torch.tensor(2, requires_grad=True); y = torch.tensor(3, requires_grad=True); z = x + y correctly sets requires_grad=True for tensors and adds them, building dynamic graph.

3. Final Answer:

   x = torch.tensor(2, requires_grad=True); y = torch.tensor(3, requires_grad=True); z = x + y -> Option D

4. Quick Check:

   requires_grad=True enables dynamic graph [OK]

Hint: Use requires_grad=True to track operations [OK]

Common Mistakes:

• Missing requires_grad=True disables graph tracking
• Using graph_static=True which is invalid
• Confusing multiplication with addition

3. What will be the output of this PyTorch code snippet?

import torch
x = torch.tensor(1.0, requires_grad=True)
y = x * 2
if y.item() > 1.5:
    z = y * 3
else:
    z = y * 4
z.backward()
print(x.grad.item())

medium

A. 3.0

B. 6.0

C. 4.0

D. 2.0

Solution

1. Step 1: Trace the computation graph

   x=1.0 with grad tracking; y = x*2 = 2.0; since y>1.5, z = y*3 = 6.0

2. Step 2: Compute gradient with backward()

   dz/dx = dz/dy * dy/dx = 3 * 2 = 6; but PyTorch accumulates gradients, so x.grad = 6

3. Final Answer:

   6.0 -> Option B

4. Quick Check:

   Gradient = 6.0 [OK]

Hint: Follow operations and chain rule for gradients [OK]

Common Mistakes:

• Ignoring the if condition effect
• Confusing gradient with output value
• Forgetting to call backward()

4. Identify the error in this PyTorch code that tries to use dynamic computation graph:

import torch
x = torch.tensor(2.0)
y = x * 3
z = y ** 2
z.backward()
print(x.grad)

medium

A. x tensor is missing requires_grad=True, so gradient is None.

B. Backward() is called on a non-scalar tensor causing error.

C. The power operator ** is not supported in PyTorch tensors.

D. The print statement syntax is incorrect.

Solution

1. Step 1: Check requires_grad flag

   x is created without requires_grad=True, so no graph tracks operations on x.

2. Step 2: Effect on backward()

   Calling backward() on z raises RuntimeError because no computation graph is tracked without requires_grad=True on x.

3. Final Answer:

   x tensor is missing requires_grad=True, so gradient is None. -> Option A

4. Quick Check:

   requires_grad=True needed for gradients [OK]

Hint: Always set requires_grad=True to track gradients [OK]

Common Mistakes:

• Assuming backward() works without requires_grad
• Thinking ** operator is invalid
• Confusing print syntax errors

5. You want to build a PyTorch model that changes its forward pass based on input size dynamically. Which feature of dynamic computation graphs helps achieve this?

hard

A. Fixing the model architecture to a single shape for all inputs.

B. Pre-compiling the entire model graph before training starts.

C. Ability to define the computation graph on-the-fly during each forward pass.

D. Using static graph optimizations for faster inference.

Solution

1. Step 1: Understand dynamic graph flexibility

   Dynamic graphs let you build the computation graph during each forward pass, adapting to input size.

2. Step 2: Contrast with static graphs

   Static graphs require fixed architecture, so they can't adapt dynamically to input changes.

3. Final Answer:

   Ability to define the computation graph on-the-fly during each forward pass. -> Option C

4. Quick Check:

   Dynamic graph = on-the-fly graph building [OK]

Hint: Dynamic graphs build model each forward pass [OK]

Common Mistakes:

• Confusing dynamic with static graph behavior
• Thinking model must be fixed before training
• Assuming static graphs allow input size changes