Custom metrics in TensorFlow

Introduction

Custom metrics help you measure exactly what matters for your model. They let you track special goals beyond built-in scores.

You want to check how well your model predicts a specific class.

You need a special score like F1 or precision not included by default.

You want to track a business-related metric during training.

You want to compare model performance with a custom rule.

You want to debug your model by watching a custom value.

Syntax

TensorFlow

def custom_metric(y_true, y_pred):
    # calculate metric value
    return metric_value

The function takes y_true (real labels) and y_pred (model predictions).

Return a single tensor value representing the metric.

Examples

This metric returns the average of all predicted values.

TensorFlow

def mean_pred(y_true, y_pred):
    return tf.reduce_mean(y_pred)

This metric calculates accuracy for binary classification by rounding predictions.

TensorFlow

def binary_accuracy(y_true, y_pred):
    y_pred_labels = tf.round(y_pred)
    return tf.reduce_mean(tf.cast(tf.equal(y_true, y_pred_labels), tf.float32))

Sample Model

This example shows how to create a custom metric for mean absolute error (MAE). The model is trained on simple data, and the custom MAE is printed after training.

TensorFlow

import tensorflow as tf

# Define a custom metric: mean absolute error manually
@tf.function
def custom_mae(y_true, y_pred):
    return tf.reduce_mean(tf.abs(y_true - y_pred))

# Create a simple model
model = tf.keras.Sequential([
    tf.keras.layers.Dense(1, input_shape=(1,))
])

# Compile model with custom metric
model.compile(optimizer='sgd', loss='mse', metrics=[custom_mae])

# Sample data
x = tf.constant([[1.0], [2.0], [3.0], [4.0]])
y = tf.constant([[2.0], [4.0], [6.0], [8.0]])

# Train model
history = model.fit(x, y, epochs=3, verbose=0)

# Evaluate model
loss, mae = model.evaluate(x, y, verbose=0)

print(f"Loss: {loss:.4f}")
print(f"Custom MAE: {mae:.4f}")

Important Notes

Custom metrics must use TensorFlow operations to work well with training.

Use @tf.function decorator for better performance but it's optional.

Custom metrics appear in training logs just like built-in metrics.

Summary

Custom metrics let you track special scores during model training.

They take true labels and predictions, then return a single number.

Use TensorFlow ops inside custom metric functions for compatibility.

Practice

1. What is the main purpose of a custom metric in TensorFlow model training?

easy

A. To track a special score based on true labels and predictions during training

B. To change the model's architecture automatically

C. To speed up the training process by using GPU

D. To save the model weights after each epoch

Solution

1. Step 1: Understand what metrics do

   Metrics measure how well the model performs by comparing true labels and predictions.

2. Step 2: Identify the role of custom metrics

   Custom metrics let you define your own way to measure performance during training.

3. Final Answer:

   To track a special score based on true labels and predictions during training -> Option A

4. Quick Check:

   Custom metric = special score tracking [OK]

Hint: Custom metrics measure model performance uniquely during training [OK]

Common Mistakes:

• Confusing metrics with model architecture changes
• Thinking metrics speed up training
• Believing metrics save model weights

2. Which of the following is the correct way to define a simple custom metric function in TensorFlow?

easy

A. def my_metric(y_true, y_pred): print(y_true, y_pred)

B. def my_metric(y_true, y_pred): return np.mean(np.abs(y_true - y_pred))

C. def my_metric(y_true, y_pred): return y_true + y_pred

D. def my_metric(y_true, y_pred): return tf.reduce_mean(tf.abs(y_true - y_pred))

Solution

1. Step 1: Check TensorFlow compatibility

   Custom metrics must use TensorFlow operations like tf.reduce_mean for compatibility.

2. Step 2: Identify correct function

   def my_metric(y_true, y_pred): return tf.reduce_mean(tf.abs(y_true - y_pred)) uses tf.reduce_mean and tf.abs correctly; def my_metric(y_true, y_pred): return np.mean(np.abs(y_true - y_pred)) uses numpy which is incompatible.

3. Final Answer:

   def my_metric(y_true, y_pred): return tf.reduce_mean(tf.abs(y_true - y_pred)) -> Option D

4. Quick Check:

   Use TensorFlow ops in custom metrics [OK]

Hint: Use TensorFlow ops, not numpy, inside custom metric functions [OK]

Common Mistakes:

• Using numpy functions inside metric causing errors
• Returning tensors without reduction to scalar
• Printing values instead of returning metric

3. Given the following custom metric code, what will be the output metric value for y_true = [1, 2, 3] and y_pred = [1, 2, 4]?

def custom_metric(y_true, y_pred):
    diff = tf.abs(y_true - y_pred)
    return tf.reduce_mean(diff)

medium

A. 1.0

B. 0.6667

C. 0.3333

D. 2.0

Solution

1. Step 1: Calculate absolute differences

   Differences: |1-1|=0, |2-2|=0, |3-4|=1

2. Step 2: Compute mean of differences

   Mean = (0 + 0 + 1) / 3 = 0.3333

3. Final Answer:

   0.3333 -> Option C

4. Quick Check:

   Mean absolute difference = 0.3333 [OK]

Hint: Calculate absolute differences, then average them [OK]

Common Mistakes:

• Summing differences without averaging
• Using squared difference instead of absolute
• Mixing up y_true and y_pred order

4. Identify the error in this custom metric function and choose the correct fix:

def faulty_metric(y_true, y_pred):
    return tf.reduce_sum(y_true - y_pred)

medium

A. Convert y_true and y_pred to numpy arrays first

B. Use tf.reduce_mean instead of tf.reduce_sum to get average error

C. Replace subtraction with addition

D. Add print statements to debug values

Solution

1. Step 1: Understand metric output

   Metrics should return a scalar representing average error, not total sum.

2. Step 2: Fix by averaging

   Replacing tf.reduce_sum with tf.reduce_mean returns average difference, a better metric.

3. Final Answer:

   Use tf.reduce_mean instead of tf.reduce_sum to get average error -> Option B

4. Quick Check:

   Average error metric needs reduce_mean [OK]

Hint: Use reduce_mean for average, not reduce_sum [OK]

Common Mistakes:

• Using reduce_sum causing large values
• Mixing TensorFlow and numpy operations
• Not returning a scalar metric

5. You want to create a custom metric that counts how many predictions differ from true labels by more than 0.5. Which of the following implementations is correct?

hard

A. def count_large_diff(y_true, y_pred):
diff = tf.abs(y_true - y_pred)
return tf.reduce_sum(tf.cast(diff > 0.5, tf.float32))

B. def count_large_diff(y_true, y_pred):
diff = y_true - y_pred
return tf.reduce_mean(diff > 0.5)

C. def count_large_diff(y_true, y_pred):
diff = tf.abs(y_true - y_pred)
return tf.reduce_sum(diff > 0.5)

D. def count_large_diff(y_true, y_pred):
diff = tf.abs(y_true - y_pred)
return tf.reduce_sum(tf.int32(diff > 0.5))

Solution

1. Step 1: Identify correct comparison and casting

   We must compare differences with 0.5 and cast boolean to float for summation.

2. Step 2: Check TensorFlow functions used

   def count_large_diff(y_true, y_pred): diff = tf.abs(y_true - y_pred) return tf.reduce_sum(tf.cast(diff > 0.5, tf.float32)) uses tf.abs, comparison, tf.cast to float32, and tf.reduce_sum correctly.

3. Final Answer:

   def count_large_diff(y_true, y_pred): diff = tf.abs(y_true - y_pred) return tf.reduce_sum(tf.cast(diff > 0.5, tf.float32)) -> Option A

4. Quick Check:

   Cast booleans to float before summing [OK]

Hint: Cast boolean mask to float before summing [OK]

Common Mistakes:

• Not casting boolean mask before summing
• Using reduce_mean instead of reduce_sum
• Using wrong data type cast like tf.int32