GradientTape for automatic differentiation in TensorFlow - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of GradientTape with simple function

What is the output of the following TensorFlow code that uses GradientTape to compute the derivative of y = x² at x = 3?

TensorFlow

import tensorflow as tf
x = tf.Variable(3.0)
with tf.GradientTape() as tape:
    y = x ** 2
dy_dx = tape.gradient(y, x)
print(dy_dx.numpy())

A 9.0

B 6.0

C 3.0

D None

💡 Hint

Recall the derivative of x squared is 2x.

The derivative of y = x² is dy/dx = 2x. At x = 3, dy/dx = 6.0.

❓ Model Choice

intermediate

Choosing correct GradientTape usage for multiple variables

You want to compute gradients of z = x * y + y² with respect to both x and y using TensorFlow GradientTape. Which code snippet correctly computes both gradients?

A

x = tf.Variable(2.0)
y = tf.Variable(3.0)
with tf.GradientTape() as tape:
    z = x * y + y ** 2
grads = tape.gradient(z, [x, y])

B

x = tf.Variable(2.0)
y = tf.Variable(3.0)
with tf.GradientTape() as tape:
    z = x * y + y ** 2
grads = tape.gradient(z, x)

C

x = tf.Variable(2.0)
y = tf.Variable(3.0)
with tf.GradientTape() as tape:
    z = x * y + y ** 2
grads = tape.gradient(z, y)

D

x = tf.Variable(2.0)
y = tf.Variable(3.0)
with tf.GradientTape() as tape:
    z = x * y + y ** 2
    grads = tape.gradient(z, [x])

💡 Hint

To get gradients for multiple variables, pass them as a list to gradient().

Passing [x, y] to tape.gradient returns gradients with respect to both variables. Passing only x or y returns gradient for that variable only.

❓ Hyperparameter

advanced

Effect of persistent=True in GradientTape

What is the effect of setting persistent=True when creating a tf.GradientTape?

A Allows multiple calls to gradient() on the same tape instance

B Improves performance by caching gradients automatically

C Prevents the tape from recording operations

D Automatically resets variables after computing gradients

💡 Hint

By default, GradientTape is deleted after one gradient call.

persistent=True keeps the tape active after calling gradient(), allowing multiple gradient computations from the same tape.

🔧 Debug

advanced

Identifying error in GradientTape usage

What error will this TensorFlow code raise and why? import tensorflow as tf x = tf.Variable(1.0) with tf.GradientTape() as tape: y = x * 2 print(tape.gradient(y, x)) print(tape.gradient(y, x))

A TypeError: gradient() expects a list of variables

B ValueError: y is not differentiable with respect to x

C No error, prints the same gradient twice

D RuntimeError: GradientTape has already been used and is not persistent

💡 Hint

GradientTape is deleted after first gradient call unless persistent=True.

By default, GradientTape is deleted after one call to gradient(), so the second call raises RuntimeError.

🧠 Conceptual

expert

Why use GradientTape for higher-order derivatives?

Why is tf.GradientTape useful for computing higher-order derivatives (like second derivatives) in TensorFlow?

A Because it stores all variable values permanently for reuse

B Because it automatically computes all derivatives up to any order without extra code

C Because it can record operations inside another GradientTape to compute gradients of gradients

D Because it uses symbolic differentiation instead of automatic differentiation

💡 Hint

Think about nesting GradientTape contexts.

To compute second derivatives, you nest GradientTape contexts so the inner tape records operations for the first derivative, and the outer tape computes gradients of those gradients.

Practice

1. What is the main purpose of TensorFlow's GradientTape?

easy

A. To automatically compute derivatives of operations for optimization

B. To visualize the model architecture

C. To load datasets for training

D. To save the trained model to disk

Solution

1. Step 1: Understand what GradientTape records

   GradientTape records operations on tensors to compute gradients automatically.

2. Step 2: Connect gradients to optimization

   These gradients help update model parameters during training to minimize loss.

3. Final Answer:

   To automatically compute derivatives of operations for optimization -> Option A

4. Quick Check:

   GradientTape = automatic differentiation [OK]

Hint: GradientTape records ops to find derivatives fast [OK]

Common Mistakes:

• Thinking GradientTape is for data loading
• Confusing it with model saving functions
• Assuming it visualizes model structure

2. Which of the following is the correct way to start recording operations with GradientTape in TensorFlow?

easy

A. GradientTape.record()

B. tf.GradientTape.start()

C. with tf.GradientTape() as tape:

D. tf.GradientTape.begin()

Solution

1. Step 1: Recall the syntax for using GradientTape

   GradientTape is used as a context manager with the with statement.

2. Step 2: Identify the correct syntax

   The correct syntax is with tf.GradientTape() as tape: to start recording.

3. Final Answer:

   with tf.GradientTape() as tape: -> Option C

4. Quick Check:

   Use 'with' block for GradientTape [OK]

Hint: GradientTape always used inside a 'with' block [OK]

Common Mistakes:

• Trying to call start() or begin() methods which don't exist
• Not using 'with' statement
• Using incorrect method names like record()

3. What will be the output of this code snippet?

import tensorflow as tf
x = tf.constant(3.0)
with tf.GradientTape() as tape:
    tape.watch(x)
    y = x * x + 2 * x + 1
dy_dx = tape.gradient(y, x)
print(dy_dx.numpy())

medium

A. 9.0

B. 11.0

C. 5.0

D. 8.0

Solution

1. Step 1: Analyze the function

   The code defines y = x*x + 2*x + 1 with x = tf.constant(3.0).

2. Step 2: Compute the derivative

   The derivative dy/dx = 2*x + 2. At x=3.0, this is 2*3 + 2 = 8.0.

3. Step 3: Verify code execution

   tape.watch(x) enables gradient tracking. tape.gradient(y, x) returns 8.0, printed via .numpy().

4. Final Answer:

   8.0 -> Option D

5. Quick Check:

   2*3 + 2 = 8.0 [OK]

Hint: Derivative of x²+2x+1 is 2x+2; plug in x=3 [OK]

Common Mistakes:

• Forgetting to watch the tensor x
• Miscomputing the derivative formula
• Confusing y with x in gradient call

4. Identify the error in this TensorFlow code using GradientTape:

import tensorflow as tf
x = tf.constant(2.0)
with tf.GradientTape() as tape:
    y = x * x
dy_dx = tape.gradient(y, x)
print(dy_dx.numpy())

medium

A. x is not watched, so gradient is None

B. GradientTape must be used without 'with' block

C. y should be defined outside the tape block

D. tape.gradient requires a list of variables

Solution

1. Step 1: Check if x is watched

   x is a constant and not watched by default, so gradient will be None.

2. Step 2: Understand how to fix

   Use tape.watch(x) to track x for gradient calculation.

3. Final Answer:

   x is not watched, so gradient is None -> Option A

4. Quick Check:

   Must watch constants for gradients [OK]

Hint: Watch constants explicitly to get gradients [OK]

Common Mistakes:

• Assuming constants are watched automatically
• Misusing GradientTape without 'with' block
• Passing wrong arguments to gradient()

5. You want to compute gradients of a function f(x, y) = x^2 * y + y^3 with respect to both x and y. Which code snippet correctly computes both gradients using TensorFlow's GradientTape?

hard

A. with tf.GradientTape() as tape:
x = tf.constant(2.0)
y = tf.constant(3.0)
f = x**2 * y + y**3
grads = tape.gradient(f, x, y)
print(grads)

B. with tf.GradientTape() as tape:
x = tf.constant(2.0)
y = tf.constant(3.0)
tape.watch([x, y])
f = x**2 * y + y**3
grads = tape.gradient(f, [x, y])
print(grads)

C. with tf.GradientTape() as tape:
x = tf.constant(2.0)
y = tf.constant(3.0)
tape.watch(x)
f = x**2 * y + y**3
grads = tape.gradient(f, y)
print(grads)

D. with tf.GradientTape() as tape:
x = tf.constant(2.0)
y = tf.constant(3.0)
tape.watch(x)
tape.watch(y)
f = x**2 * y + y**3
grads = tape.gradient(f, [x, y])
print(grads)

Solution

1. Step 1: Understand watching variables

   Constants must be watched explicitly to compute gradients.

2. Step 2: Check gradient call syntax

   GradientTape.gradient expects a list of variables to compute gradients for multiple variables.

3. Step 3: Compare options

   with tf.GradientTape() as tape: x = tf.constant(2.0) y = tf.constant(3.0) tape.watch([x, y]) f = x**2 * y + y**3 grads = tape.gradient(f, [x, y]) print(grads) calls tape.watch([x, y]) once with a list, then passes [x, y] to gradient().

4. Final Answer:

   Explicitly watch both x and y together and pass [x, y] to tape.gradient(f, [x, y]) -> Option B

5. Quick Check:

   Watch all variables as list and pass as list to gradient() [OK]

Hint: Watch all variables and pass as list to gradient() [OK]

Common Mistakes:

• Not watching all variables
• Passing variables separately instead of as list
• Computing gradient for only one variable