Custom metrics in TensorFlow - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of a custom accuracy metric in TensorFlow

Consider the following TensorFlow custom metric function that calculates accuracy by comparing rounded predictions to true labels. What will be the printed output after running the code?

TensorFlow

import tensorflow as tf
import numpy as np

class CustomAccuracy(tf.keras.metrics.Metric):
    def __init__(self, name='custom_accuracy', **kwargs):
        super().__init__(name=name, **kwargs)
        self.total = self.add_weight(name='total', initializer='zeros')
        self.count = self.add_weight(name='count', initializer='zeros')

    def update_state(self, y_true, y_pred, sample_weight=None):
        y_pred_rounded = tf.round(y_pred)
        values = tf.cast(tf.equal(y_true, y_pred_rounded), tf.float32)
        self.total.assign_add(tf.reduce_sum(values))
        self.count.assign_add(tf.cast(tf.size(y_true), tf.float32))

    def result(self):
        return self.total / self.count

    def reset_states(self):
        self.total.assign(0)
        self.count.assign(0)

metric = CustomAccuracy()
y_true = tf.constant([0, 1, 1, 0], dtype=tf.float32)
y_pred = tf.constant([0.1, 0.8, 0.4, 0.3], dtype=tf.float32)
metric.update_state(y_true, y_pred)
print(f"Custom accuracy: {metric.result().numpy():.2f}")

A Custom accuracy: 0.75

B Custom accuracy: 0.50

C Custom accuracy: 1.00

D Custom accuracy: 0.25

💡 Hint

Think about how rounding predictions affects the comparison to true labels.

The predictions are rounded: [0.1 -> 0, 0.8 -> 1, 0.4 -> 0, 0.3 -> 0]. Comparing to true labels [0,1,1,0], matches occur at positions 0,1, and 3, so 3 correct out of 4 total, resulting in 0.75 accuracy.

❓ Model Choice

intermediate

Choosing the correct custom metric for regression error

You want to create a custom metric in TensorFlow to measure the mean absolute error (MAE) during training of a regression model. Which of the following custom metric implementations correctly computes MAE?

A

class CustomMAE(tf.keras.metrics.Metric):
    def __init__(self):
        super().__init__()
        self.total_error = self.add_weight(name='total_error', initializer='zeros')
    def update_state(self, y_true, y_pred, sample_weight=None):
        error = tf.abs(y_true - y_pred)
        self.total_error.assign_add(tf.reduce_mean(error))
    def result(self):
        return self.total_error

B

class CustomMAE(tf.keras.metrics.Metric):
    def __init__(self):
        super().__init__()
        self.total_error = self.add_weight(name='total_error', initializer='zeros')
        self.count = self.add_weight(name='count', initializer='zeros')
    def update_state(self, y_true, y_pred, sample_weight=None):
        error = tf.abs(y_true - y_pred)
        self.total_error.assign_add(tf.reduce_sum(error))
        self.count.assign_add(tf.cast(tf.size(y_true), tf.float32))
    def result(self):
        return self.total_error / self.count

C

class CustomMAE(tf.keras.metrics.Metric):
    def __init__(self):
        super().__init__()
        self.total_error = self.add_weight(name='total_error', initializer='zeros')
        self.count = self.add_weight(name='count', initializer='zeros')
    def update_state(self, y_true, y_pred, sample_weight=None):
        error = tf.square(y_true - y_pred)
        self.total_error.assign_add(tf.reduce_sum(error))
        self.count.assign_add(tf.cast(tf.size(y_true), tf.float32))
    def result(self):
        return tf.sqrt(self.total_error / self.count)

D

class CustomMAE(tf.keras.metrics.Metric):
    def __init__(self):
        super().__init__()
        self.total_error = self.add_weight(name='total_error', initializer='zeros')
        self.count = self.add_weight(name='count', initializer='zeros')
    def update_state(self, y_true, y_pred, sample_weight=None):
        error = y_true - y_pred
        self.total_error.assign_add(tf.reduce_sum(error))
        self.count.assign_add(tf.cast(tf.size(y_true), tf.float32))
    def result(self):
        return self.total_error / self.count

💡 Hint

MAE is the average of absolute differences between true and predicted values.

Option B correctly sums absolute errors and divides by count to get mean absolute error. Option B sums means repeatedly, inflating error. Option B computes RMSE, not MAE. Option B sums raw errors, which can cancel out.

❓ Hyperparameter

advanced

Effect of batch size on custom metric calculation

You implemented a custom metric in TensorFlow that accumulates values over batches during training. If you increase the batch size significantly, what is the most likely effect on the metric's reported value after one epoch?

A The metric value will always increase proportionally with batch size.

B The metric value becomes less accurate because larger batches cause overflow in the accumulation variables.

C The metric value remains the same because it accumulates totals correctly regardless of batch size.

D The metric value may change because the number of update calls decreases, affecting floating point precision accumulation.

💡 Hint

Think about how floating point addition and number of updates affect accumulated sums.

Changing batch size changes how many times update_state is called. Fewer calls with larger batches can cause slight differences due to floating point precision, but the metric should still be correct overall. Overflow is unlikely with typical batch sizes.

🔧 Debug

advanced

Identifying error in custom metric implementation

Examine the following custom metric code. What error will occur when this metric is used during model training?

TensorFlow

import tensorflow as tf

class FaultyMetric(tf.keras.metrics.Metric):
    def __init__(self):
        super().__init__()
        self.total = 0
        self.count = 0

    def update_state(self, y_true, y_pred, sample_weight=None):
        values = tf.cast(tf.equal(y_true, tf.round(y_pred)), tf.float32)
        self.total += tf.reduce_sum(values)
        self.count += tf.size(y_true)

    def result(self):
        return self.total / self.count

    def reset_states(self):
        self.total = 0
        self.count = 0

A TypeError because self.total and self.count are Python ints, not TensorFlow variables.

B No error; the metric works correctly.

C ValueError because tf.equal cannot compare tensors of different shapes.

D AttributeError because reset_states method is misspelled.

💡 Hint

Check how variables are initialized and updated inside a TensorFlow metric.

TensorFlow metrics require state variables created with add_weight to track values across batches. Using plain Python ints and += causes TypeError when adding tensors to ints.

🧠 Conceptual

expert

Why use custom metrics instead of built-in metrics?

Which of the following is the best reason to implement a custom metric in TensorFlow instead of using built-in metrics?

A To avoid using TensorFlow's internal state management and simplify metric calculations.

B Because built-in metrics are slower and less efficient than any custom metric implementation.

C To measure a specific performance aspect not covered by built-in metrics, tailored to your problem's unique needs.

D Because custom metrics automatically improve model accuracy without changing the model.

💡 Hint

Think about why you might want to track something special during training.

Custom metrics allow tracking of problem-specific performance measures that built-in metrics do not provide. Built-in metrics are optimized and reliable. Custom metrics do not inherently improve accuracy or simplify state management.

Practice

1. What is the main purpose of a custom metric in TensorFlow model training?

easy

A. To track a special score based on true labels and predictions during training

B. To change the model's architecture automatically

C. To speed up the training process by using GPU

D. To save the model weights after each epoch

Solution

1. Step 1: Understand what metrics do

   Metrics measure how well the model performs by comparing true labels and predictions.

2. Step 2: Identify the role of custom metrics

   Custom metrics let you define your own way to measure performance during training.

3. Final Answer:

   To track a special score based on true labels and predictions during training -> Option A

4. Quick Check:

   Custom metric = special score tracking [OK]

Hint: Custom metrics measure model performance uniquely during training [OK]

Common Mistakes:

• Confusing metrics with model architecture changes
• Thinking metrics speed up training
• Believing metrics save model weights

2. Which of the following is the correct way to define a simple custom metric function in TensorFlow?

easy

A. def my_metric(y_true, y_pred): print(y_true, y_pred)

B. def my_metric(y_true, y_pred): return np.mean(np.abs(y_true - y_pred))

C. def my_metric(y_true, y_pred): return y_true + y_pred

D. def my_metric(y_true, y_pred): return tf.reduce_mean(tf.abs(y_true - y_pred))

Solution

1. Step 1: Check TensorFlow compatibility

   Custom metrics must use TensorFlow operations like tf.reduce_mean for compatibility.

2. Step 2: Identify correct function

   def my_metric(y_true, y_pred): return tf.reduce_mean(tf.abs(y_true - y_pred)) uses tf.reduce_mean and tf.abs correctly; def my_metric(y_true, y_pred): return np.mean(np.abs(y_true - y_pred)) uses numpy which is incompatible.

3. Final Answer:

   def my_metric(y_true, y_pred): return tf.reduce_mean(tf.abs(y_true - y_pred)) -> Option D

4. Quick Check:

   Use TensorFlow ops in custom metrics [OK]

Hint: Use TensorFlow ops, not numpy, inside custom metric functions [OK]

Common Mistakes:

• Using numpy functions inside metric causing errors
• Returning tensors without reduction to scalar
• Printing values instead of returning metric

3. Given the following custom metric code, what will be the output metric value for y_true = [1, 2, 3] and y_pred = [1, 2, 4]?

def custom_metric(y_true, y_pred):
    diff = tf.abs(y_true - y_pred)
    return tf.reduce_mean(diff)

medium

A. 1.0

B. 0.6667

C. 0.3333

D. 2.0

Solution

1. Step 1: Calculate absolute differences

   Differences: |1-1|=0, |2-2|=0, |3-4|=1

2. Step 2: Compute mean of differences

   Mean = (0 + 0 + 1) / 3 = 0.3333

3. Final Answer:

   0.3333 -> Option C

4. Quick Check:

   Mean absolute difference = 0.3333 [OK]

Hint: Calculate absolute differences, then average them [OK]

Common Mistakes:

• Summing differences without averaging
• Using squared difference instead of absolute
• Mixing up y_true and y_pred order

4. Identify the error in this custom metric function and choose the correct fix:

def faulty_metric(y_true, y_pred):
    return tf.reduce_sum(y_true - y_pred)

medium

A. Convert y_true and y_pred to numpy arrays first

B. Use tf.reduce_mean instead of tf.reduce_sum to get average error

C. Replace subtraction with addition

D. Add print statements to debug values

Solution

1. Step 1: Understand metric output

   Metrics should return a scalar representing average error, not total sum.

2. Step 2: Fix by averaging

   Replacing tf.reduce_sum with tf.reduce_mean returns average difference, a better metric.

3. Final Answer:

   Use tf.reduce_mean instead of tf.reduce_sum to get average error -> Option B

4. Quick Check:

   Average error metric needs reduce_mean [OK]

Hint: Use reduce_mean for average, not reduce_sum [OK]

Common Mistakes:

• Using reduce_sum causing large values
• Mixing TensorFlow and numpy operations
• Not returning a scalar metric

5. You want to create a custom metric that counts how many predictions differ from true labels by more than 0.5. Which of the following implementations is correct?

hard

A. def count_large_diff(y_true, y_pred):
diff = tf.abs(y_true - y_pred)
return tf.reduce_sum(tf.cast(diff > 0.5, tf.float32))

B. def count_large_diff(y_true, y_pred):
diff = y_true - y_pred
return tf.reduce_mean(diff > 0.5)

C. def count_large_diff(y_true, y_pred):
diff = tf.abs(y_true - y_pred)
return tf.reduce_sum(diff > 0.5)

D. def count_large_diff(y_true, y_pred):
diff = tf.abs(y_true - y_pred)
return tf.reduce_sum(tf.int32(diff > 0.5))

Solution

1. Step 1: Identify correct comparison and casting

   We must compare differences with 0.5 and cast boolean to float for summation.

2. Step 2: Check TensorFlow functions used

   def count_large_diff(y_true, y_pred): diff = tf.abs(y_true - y_pred) return tf.reduce_sum(tf.cast(diff > 0.5, tf.float32)) uses tf.abs, comparison, tf.cast to float32, and tf.reduce_sum correctly.

3. Final Answer:

   def count_large_diff(y_true, y_pred): diff = tf.abs(y_true - y_pred) return tf.reduce_sum(tf.cast(diff > 0.5, tf.float32)) -> Option A

4. Quick Check:

   Cast booleans to float before summing [OK]

Hint: Cast boolean mask to float before summing [OK]

Common Mistakes:

• Not casting boolean mask before summing
• Using reduce_mean instead of reduce_sum
• Using wrong data type cast like tf.int32