Custom training step in TensorFlow - Model Metrics & Evaluation

Metrics & Evaluation - Custom training step

Which metric matters for Custom training step and WHY

When you create a custom training step, you control how the model learns. The key metrics to watch are loss and accuracy (or other task-specific metrics like precision or recall). Loss tells you how far off the model's predictions are from the true answers. Accuracy or other metrics show how well the model is performing on the task. Monitoring these during training helps you know if your custom step is helping the model learn better or not.

Confusion matrix example

For classification tasks, a confusion matrix helps understand errors your model makes during training with a custom step. Here is an example for a binary classifier:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP) = 50 | False Negative (FN) = 10 |
      | False Positive (FP) = 5 | True Negative (TN) = 35 |
    

Total samples = 50 + 10 + 5 + 35 = 100

From this, you can calculate precision, recall, and F1 score to evaluate your custom training step's effect.

Precision vs Recall tradeoff with examples

In a custom training step, you might change how the model balances precision and recall.

• Precision = TP / (TP + FP): How many predicted positives are actually correct?
• Recall = TP / (TP + FN): How many actual positives did the model find?

Example:

• Spam filter: High precision is important to avoid marking good emails as spam.
• Cancer detection: High recall is critical to catch as many cancer cases as possible.

Your custom training step can be designed to improve one metric at the cost of the other, depending on your goal.

What "good" vs "bad" metric values look like for Custom training step

Good metrics after a custom training step:

• Loss steadily decreases over epochs.
• Accuracy or task metric improves consistently.
• Precision and recall are balanced or meet your task needs.

Bad metrics might show:

• Loss stays high or fluctuates wildly.
• Accuracy does not improve or gets worse.
• Precision or recall is very low, indicating poor predictions.

Common pitfalls in metrics with Custom training step

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced.
• Data leakage: If training data leaks into validation, metrics look better but model won't generalize.
• Overfitting: Training metrics improve but validation metrics worsen, showing the model memorizes data.
• Incorrect metric calculation: Custom steps must correctly update metrics each batch.

Self-check question

Your model has 98% accuracy but only 12% recall on fraud detection after a custom training step. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses most fraud cases, which is dangerous. Even though accuracy is high, it likely predicts most samples as non-fraud. For fraud detection, high recall is critical to catch fraud cases.

Key Result

Loss and task-specific metrics like accuracy, precision, and recall are key to evaluate custom training steps effectively.

Practice

1.

What is the main purpose of a custom training step in TensorFlow?

easy

A. To visualize model predictions during training

B. To automatically generate a model architecture

C. To load datasets from external sources

D. To define exactly how the model learns from each batch of data

Solution

1. Step 1: Understand the role of a training step

   A training step controls how the model updates its weights using data batches.

2. Step 2: Identify what custom training step changes

   Custom training steps let you write your own logic for learning, including gradient calculation and weight updates.

3. Final Answer:

   To define exactly how the model learns from each batch of data -> Option D

4. Quick Check:

   Custom training step = control learning process [OK]

Hint: Custom training step controls learning per batch [OK]

Common Mistakes:

• Thinking it builds model architecture
• Confusing with data loading
• Assuming it only visualizes outputs

2.

Which of the following is the correct way to start recording operations for gradient calculation in a custom training step?

def train_step(inputs):
    # What goes here?
    ...

easy

A. with tf.Variable() as var:

B. with tf.GradientTape() as tape:

C. with tf.Session() as sess:

D. with tf.data.Dataset() as ds:

Solution

1. Step 1: Recall TensorFlow's gradient recording method

   TensorFlow uses tf.GradientTape() to record operations for automatic differentiation.

2. Step 2: Match correct syntax for gradient recording

   The correct syntax to start recording is with tf.GradientTape() as tape:.

3. Final Answer:

   with tf.GradientTape() as tape: -> Option B

4. Quick Check:

   Gradient recording uses tf.GradientTape [OK]

Hint: Use tf.GradientTape() to record gradients [OK]

Common Mistakes:

• Using tf.Session() which is for old TF versions
• Confusing tf.Variable() with gradient recording
• Using tf.data.Dataset() incorrectly here

3.

What will be the output of the following custom training step code snippet?

import tensorflow as tf

model = tf.keras.Sequential([tf.keras.layers.Dense(1)])
optimizer = tf.keras.optimizers.SGD(learning_rate=0.1)

@tf.function
def train_step(x, y):
    with tf.GradientTape() as tape:
        predictions = model(x, training=True)
        loss = tf.reduce_mean(tf.square(y - predictions))
    gradients = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
    return loss

x = tf.constant([[1.0], [2.0]])
y = tf.constant([[3.0], [5.0]])
loss_value = train_step(x, y).numpy()
print(round(loss_value, 2))

medium

A. 1.0

B. 9.0

C. 4.0

D. 0.0

Solution

1. Step 1: Calculate initial predictions and loss

   Model starts with random weights near zero, so predictions ~0. Loss = mean squared error between y and predictions = mean of [9, 25] = 17.

2. Step 2: After one gradient update with learning rate 0.1, loss reduces

   One step reduces loss roughly to around 4.0 (not zero yet).

3. Final Answer:

   4.0 -> Option C

4. Quick Check:

   One step reduces loss from ~17 to ~4 [OK]

Hint: One step reduces loss but not to zero [OK]

Common Mistakes:

• Assuming loss becomes zero immediately
• Ignoring initial random weights
• Confusing loss with prediction values

4.

Identify the error in this custom training step code and select the fix:

def train_step(x, y):
    with tf.GradientTape() as tape:
        predictions = model(x)
        loss = tf.reduce_mean(tf.square(y - predictions))
    gradients = tape.gradient(loss)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
    return loss

medium

A. Pass model.trainable_variables as second argument to tape.gradient

B. Remove tf.GradientTape() context manager

C. Change loss calculation to tf.reduce_sum

D. Call optimizer.minimize instead of apply_gradients

Solution

1. Step 1: Check tape.gradient usage

   tape.gradient requires two arguments: loss and variables to differentiate with respect to.

2. Step 2: Identify missing argument

   Code misses the second argument (model.trainable_variables), causing error.

3. Final Answer:

   Pass model.trainable_variables as second argument to tape.gradient -> Option A

4. Quick Check:

   tape.gradient(loss, variables) needs variables [OK]

Hint: Always pass variables to tape.gradient [OK]

Common Mistakes:

• Forgetting second argument in tape.gradient
• Removing GradientTape context
• Changing loss function unnecessarily

5.

You want to track both loss and accuracy in a custom training step for a classification model. Which code snippet correctly updates and returns these metrics?

loss_metric = tf.keras.metrics.Mean()
accuracy_metric = tf.keras.metrics.SparseCategoricalAccuracy()

@tf.function
def train_step(x, y):
    with tf.GradientTape() as tape:
        logits = model(x, training=True)
        loss = loss_fn(y, logits)
    gradients = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(gradients, model.trainable_variables))
    
    # What goes here?
    ...

hard

A. loss_metric.update_state(loss); accuracy_metric.update_state(y, logits); return loss_metric.result(), accuracy_metric.result()

B. loss_metric = loss; accuracy_metric = accuracy(y, logits); return loss_metric, accuracy_metric

C. loss_metric.add(loss); accuracy_metric.add(y, logits); return loss_metric, accuracy_metric

D. loss_metric.reset_states(); accuracy_metric.reset_states(); return loss_metric.result(), accuracy_metric.result()

Solution

1. Step 1: Understand metric update methods

   TensorFlow metrics use update_state() to add new values for tracking.

2. Step 2: Correctly update and return metric results

   Use loss_metric.update_state(loss) and accuracy_metric.update_state(y, logits), then return their result().

3. Final Answer:

   loss_metric.update_state(loss); accuracy_metric.update_state(y, logits); return loss_metric.result(), accuracy_metric.result() -> Option A

4. Quick Check:

   Use update_state() and result() for metrics [OK]

Hint: Use update_state() then result() to track metrics [OK]

Common Mistakes:

• Assigning metric variables instead of updating
• Using non-existent add() method
• Resetting metrics inside training step