Why Custom metrics in TensorFlow? - Purpose & Use Cases

The Big Idea

What if your model's success metric is hiding its biggest mistakes?

The Scenario

Imagine you are training a model to detect rare diseases from medical images. The usual accuracy metric doesn't tell you if the model is catching those rare cases well enough.

The Problem

Using only built-in metrics like accuracy can hide important details. You might miss how well the model performs on specific cases, leading to wrong conclusions and poor decisions.

The Solution

Custom metrics let you define exactly what success means for your problem. You can track the model's performance on the parts that matter most, making your training smarter and more focused.

Before vs After

✗ Before

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

✓ After

def custom_metric(y_true, y_pred):
    # define your own metric logic here
    metric_value = ...  # compute the metric value
    return metric_value

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=[custom_metric])

What It Enables

Custom metrics unlock the power to measure exactly what matters, guiding your model to real-world success.

Real Life Example

In fraud detection, a custom metric can focus on catching fraudulent transactions rather than just overall accuracy, helping banks stop fraud more effectively.

Key Takeaways

Built-in metrics may not fit every problem.

Custom metrics let you track what truly matters.

This leads to better, more useful models.

Practice

1. What is the main purpose of a custom metric in TensorFlow model training?

easy

A. To track a special score based on true labels and predictions during training

B. To change the model's architecture automatically

C. To speed up the training process by using GPU

D. To save the model weights after each epoch

Solution

1. Step 1: Understand what metrics do

   Metrics measure how well the model performs by comparing true labels and predictions.

2. Step 2: Identify the role of custom metrics

   Custom metrics let you define your own way to measure performance during training.

3. Final Answer:

   To track a special score based on true labels and predictions during training -> Option A

4. Quick Check:

   Custom metric = special score tracking [OK]

Hint: Custom metrics measure model performance uniquely during training [OK]

Common Mistakes:

• Confusing metrics with model architecture changes
• Thinking metrics speed up training
• Believing metrics save model weights

2. Which of the following is the correct way to define a simple custom metric function in TensorFlow?

easy

A. def my_metric(y_true, y_pred): print(y_true, y_pred)

B. def my_metric(y_true, y_pred): return np.mean(np.abs(y_true - y_pred))

C. def my_metric(y_true, y_pred): return y_true + y_pred

D. def my_metric(y_true, y_pred): return tf.reduce_mean(tf.abs(y_true - y_pred))

Solution

1. Step 1: Check TensorFlow compatibility

   Custom metrics must use TensorFlow operations like tf.reduce_mean for compatibility.

2. Step 2: Identify correct function

   def my_metric(y_true, y_pred): return tf.reduce_mean(tf.abs(y_true - y_pred)) uses tf.reduce_mean and tf.abs correctly; def my_metric(y_true, y_pred): return np.mean(np.abs(y_true - y_pred)) uses numpy which is incompatible.

3. Final Answer:

   def my_metric(y_true, y_pred): return tf.reduce_mean(tf.abs(y_true - y_pred)) -> Option D

4. Quick Check:

   Use TensorFlow ops in custom metrics [OK]

Hint: Use TensorFlow ops, not numpy, inside custom metric functions [OK]

Common Mistakes:

• Using numpy functions inside metric causing errors
• Returning tensors without reduction to scalar
• Printing values instead of returning metric

3. Given the following custom metric code, what will be the output metric value for y_true = [1, 2, 3] and y_pred = [1, 2, 4]?

def custom_metric(y_true, y_pred):
    diff = tf.abs(y_true - y_pred)
    return tf.reduce_mean(diff)

medium

A. 1.0

B. 0.6667

C. 0.3333

D. 2.0

Solution

1. Step 1: Calculate absolute differences

   Differences: |1-1|=0, |2-2|=0, |3-4|=1

2. Step 2: Compute mean of differences

   Mean = (0 + 0 + 1) / 3 = 0.3333

3. Final Answer:

   0.3333 -> Option C

4. Quick Check:

   Mean absolute difference = 0.3333 [OK]

Hint: Calculate absolute differences, then average them [OK]

Common Mistakes:

• Summing differences without averaging
• Using squared difference instead of absolute
• Mixing up y_true and y_pred order

4. Identify the error in this custom metric function and choose the correct fix:

def faulty_metric(y_true, y_pred):
    return tf.reduce_sum(y_true - y_pred)

medium

A. Convert y_true and y_pred to numpy arrays first

B. Use tf.reduce_mean instead of tf.reduce_sum to get average error

C. Replace subtraction with addition

D. Add print statements to debug values

Solution

1. Step 1: Understand metric output

   Metrics should return a scalar representing average error, not total sum.

2. Step 2: Fix by averaging

   Replacing tf.reduce_sum with tf.reduce_mean returns average difference, a better metric.

3. Final Answer:

   Use tf.reduce_mean instead of tf.reduce_sum to get average error -> Option B

4. Quick Check:

   Average error metric needs reduce_mean [OK]

Hint: Use reduce_mean for average, not reduce_sum [OK]

Common Mistakes:

• Using reduce_sum causing large values
• Mixing TensorFlow and numpy operations
• Not returning a scalar metric

5. You want to create a custom metric that counts how many predictions differ from true labels by more than 0.5. Which of the following implementations is correct?

hard

A. def count_large_diff(y_true, y_pred):
diff = tf.abs(y_true - y_pred)
return tf.reduce_sum(tf.cast(diff > 0.5, tf.float32))

B. def count_large_diff(y_true, y_pred):
diff = y_true - y_pred
return tf.reduce_mean(diff > 0.5)

C. def count_large_diff(y_true, y_pred):
diff = tf.abs(y_true - y_pred)
return tf.reduce_sum(diff > 0.5)

D. def count_large_diff(y_true, y_pred):
diff = tf.abs(y_true - y_pred)
return tf.reduce_sum(tf.int32(diff > 0.5))

Solution

1. Step 1: Identify correct comparison and casting

   We must compare differences with 0.5 and cast boolean to float for summation.

2. Step 2: Check TensorFlow functions used

   def count_large_diff(y_true, y_pred): diff = tf.abs(y_true - y_pred) return tf.reduce_sum(tf.cast(diff > 0.5, tf.float32)) uses tf.abs, comparison, tf.cast to float32, and tf.reduce_sum correctly.

3. Final Answer:

   def count_large_diff(y_true, y_pred): diff = tf.abs(y_true - y_pred) return tf.reduce_sum(tf.cast(diff > 0.5, tf.float32)) -> Option A

4. Quick Check:

   Cast booleans to float before summing [OK]

Hint: Cast boolean mask to float before summing [OK]

Common Mistakes:

• Not casting boolean mask before summing
• Using reduce_mean instead of reduce_sum
• Using wrong data type cast like tf.int32