Custom metrics in TensorFlow - Deep Dive

Overview - Custom metrics

What is it?

Custom metrics are user-defined ways to measure how well a machine learning model is doing during training and testing. Instead of using only built-in measurements like accuracy or loss, you can create your own rules to check specific behaviors or qualities of the model. This helps you understand your model better and improve it in ways that matter most for your problem.

Why it matters

Without custom metrics, you might miss important details about your model's performance that built-in metrics don't capture. For example, if you care about how well your model detects rare events or balances different types of errors, custom metrics let you track those exactly. This leads to better models that solve real problems more effectively.

Where it fits

Before learning custom metrics, you should understand basic model training, built-in metrics, and how models make predictions. After mastering custom metrics, you can explore advanced topics like custom loss functions, callbacks, and model interpretability techniques.

Mental Model

Core Idea

Custom metrics are personalized rulers you create to measure exactly what matters for your model's success.

Think of it like...

It's like baking a cake and using a special taste test you design yourself, instead of just checking if it looks good or rises well. Your taste test focuses on the flavor balance you care about most.

┌─────────────────────────────┐
│       Model Training        │
├─────────────┬───────────────┤
│ Built-in    │ Custom Metric │
│ Metrics     │ Calculation   │
│ (Accuracy,  │ (User-defined │
│ Loss, etc.) │ functions)    │
└─────────────┴───────────────┘
          ↓                 ↓
      Model Feedback    Tailored Insights

Build-Up - 7 Steps

1

Foundation

Understanding Metrics in ML

Concept: Learn what metrics are and why they matter in machine learning.

Metrics are numbers that tell us how well a model is doing. Common examples include accuracy (how many predictions are correct) and loss (how far predictions are from true values). These help us decide if the model is improving during training.

Result

You know that metrics give feedback on model quality during training and testing.

Understanding metrics is essential because they guide how we improve models and know when to stop training.

2

Foundation

Built-in Metrics in TensorFlow

Concept: Explore the default metrics TensorFlow offers and how to use them.

TensorFlow provides many ready-to-use metrics like 'accuracy', 'precision', and 'recall'. You add them when compiling your model, and TensorFlow calculates them automatically during training and evaluation.

Result

You can track common performance measures without extra coding.

Knowing built-in metrics helps you understand what custom metrics add beyond these standard measures.

3

Intermediate

Why Create Custom Metrics?

🤔Before reading on: do you think built-in metrics cover all model evaluation needs? Commit to yes or no.

Concept: Identify situations where built-in metrics are not enough and custom metrics become necessary.

Sometimes, built-in metrics don't capture what you really care about. For example, in medical diagnosis, missing a disease (false negative) is worse than a false alarm (false positive). Custom metrics let you measure these specific concerns.

Result

You understand the motivation to create custom metrics tailored to your problem.

Recognizing the limits of built-in metrics opens the door to more precise and meaningful model evaluation.

4

Intermediate

Writing a Simple Custom Metric

🤔Before reading on: do you think a custom metric must be a complex class or can it be a simple function? Commit to your answer.

Concept: Learn how to write a basic custom metric as a function in TensorFlow.

You can define a function that takes true labels and predictions, then returns a number representing your metric. For example, a metric that counts how many predictions are exactly equal to true labels: import tensorflow as tf def custom_accuracy(y_true, y_pred): y_pred_rounded = tf.round(y_pred) correct = tf.cast(tf.equal(y_true, y_pred_rounded), tf.float32) return tf.reduce_mean(correct) Then pass this function to model.compile(metrics=[custom_accuracy]).

Result

Your model reports your custom metric during training and evaluation.

Knowing that custom metrics can be simple functions lowers the barrier to creating tailored evaluations.

5

Intermediate

Using tf.keras.metrics.Metric Class

🤔Before reading on: do you think stateful metrics need special handling compared to simple functions? Commit to yes or no.

Concept: Learn how to build more complex custom metrics by subclassing TensorFlow's Metric class to keep track of state across batches.

For metrics that need to remember information across batches (like counting total correct predictions), subclass tf.keras.metrics.Metric: import tensorflow as tf class CustomAccuracy(tf.keras.metrics.Metric): def __init__(self, name='custom_accuracy', **kwargs): super().__init__(name=name, **kwargs) self.total = self.add_weight(name='total', initializer='zeros') self.count = self.add_weight(name='count', initializer='zeros') def update_state(self, y_true, y_pred, sample_weight=None): y_pred_rounded = tf.round(y_pred) matches = tf.cast(tf.equal(y_true, y_pred_rounded), tf.float32) self.total.assign_add(tf.reduce_sum(matches)) self.count.assign_add(tf.cast(tf.size(y_true), tf.float32)) def result(self): return self.total / self.count def reset_states(self): self.total.assign(0) self.count.assign(0) Use this class in model.compile(metrics=[CustomAccuracy()]).

Result

You can track metrics that accumulate information over time correctly.

Understanding stateful metrics is key for accurate evaluation when data is processed in batches.

6

Advanced

Handling Metric Behavior in Distributed Training

🤔Before reading on: do you think custom metrics automatically work correctly on multiple GPUs or TPUs? Commit to yes or no.

Concept: Learn how custom metrics behave in distributed training and what to do to ensure correct results.

In distributed training, batches are split across devices. Stateful metrics must aggregate results from all devices properly. TensorFlow handles this if you use tf.keras.metrics.Metric correctly, but if you use simple functions or custom variables, you might get wrong results. Always test metrics in your distributed setup.

Result

Your custom metrics report correct values even when training on multiple devices.

Knowing how distribution affects metric calculation prevents subtle bugs in large-scale training.

7

Expert

Optimizing Custom Metrics for Performance

🤔Before reading on: do you think custom metrics can slow down training significantly? Commit to yes or no.

Concept: Explore how to write efficient custom metrics that do not slow down training or cause memory issues.

Custom metrics run every batch, so inefficient code can slow training. Use TensorFlow operations (tf.*) instead of Python loops. Avoid creating new variables inside update_state. Use vectorized operations and keep computations simple. Profiling tools can help find bottlenecks.

Result

Your training runs smoothly with custom metrics, even on large datasets.

Understanding performance implications of custom metrics helps maintain fast and scalable training.

Under the Hood

Custom metrics in TensorFlow are functions or classes that receive true labels and model predictions each batch. They compute values using TensorFlow operations, which build a computation graph or run eagerly. Stateful metrics keep internal variables to accumulate information across batches. During training, TensorFlow calls these metrics after each batch and epoch, aggregating results and reporting them. In distributed setups, TensorFlow synchronizes metric states across devices to produce correct global values.

Why designed this way?

TensorFlow designed custom metrics to be flexible and efficient. Functions allow quick simple metrics, while the Metric class supports complex stateful metrics needed for real-world problems. This design balances ease of use and power. Alternatives like only function-based metrics would limit capability, while only class-based would be too complex for beginners.

┌───────────────────────────────┐
│       Training Loop           │
├─────────────┬─────────────────┤
│ Batch Data  │ Model Predicts  │
│ (x, y_true) │ y_pred          │
└──────┬──────┴─────────┬───────┘
       │                │
       ▼                ▼
┌───────────────┐  ┌───────────────┐
│ Metric Update │  │ Loss Compute  │
│ (update_state)│  │               │
└──────┬────────┘  └──────┬────────┘
       │                  │
       ▼                  ▼
┌─────────────────────────────────┐
│ Metric Result Aggregation (result)│
└─────────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think a custom metric must always be a class, never a simple function? Commit to yes or no.

Common Belief: Custom metrics have to be complex classes to work properly.

Reality: Custom metrics can be simple functions for many cases, making them easy to write and use.

Why it matters: Believing this can discourage beginners from creating useful custom metrics quickly.

Quick: Do you think custom metrics automatically work correctly in distributed training without extra care? Commit to yes or no.

Common Belief: Custom metrics just work the same on single and multiple devices.

Reality: Some custom metrics need special handling to aggregate results correctly across devices in distributed training.

Why it matters: Ignoring this can cause wrong metric values, misleading model evaluation in large-scale training.

Quick: Do you think custom metrics affect model training speed significantly? Commit to yes or no.

Common Belief: Custom metrics have no impact on training speed.

Reality: Inefficient custom metrics can slow down training because they run every batch.

Why it matters: Not optimizing metrics can cause longer training times and wasted resources.

Quick: Do you think custom metrics can change model weights during training? Commit to yes or no.

Common Belief: Custom metrics can influence model learning by changing weights.

Reality: Metrics only measure performance; they do not affect model weights or training directly.

Why it matters: Confusing metrics with loss functions can lead to wrong implementations and expectations.

Expert Zone

1

Custom metrics must be differentiable only if used as loss functions; otherwise, they can use any operations.

2

Resetting metric states properly between epochs is crucial to avoid accumulating stale values.

3

Using sample weights inside custom metrics requires careful handling to match TensorFlow's weighting behavior.

When NOT to use

Avoid custom metrics when a built-in metric already perfectly fits your needs, as custom code can introduce bugs or slowdowns. For complex evaluation, consider post-training analysis tools or callbacks instead of metrics during training.

Production Patterns

In production, custom metrics often monitor business-specific KPIs like conversion rates or error costs. They are integrated with monitoring dashboards and alerting systems to track model health continuously.

Connections

Custom Loss Functions

Builds-on

Understanding custom metrics helps grasp custom loss functions, as both involve defining computations on predictions and labels but serve different roles: metrics evaluate, losses guide training.

Software Testing Metrics

Similar pattern

Both custom metrics in ML and software testing metrics measure quality tailored to specific goals, showing how measurement customization is key in many fields.

Quality Control in Manufacturing

Analogous process

Just like custom metrics tailor checks to product features in manufacturing, ML custom metrics tailor evaluation to model goals, highlighting universal principles of targeted quality measurement.

Common Pitfalls

#1 Writing a custom metric as a Python function with side effects or non-TensorFlow operations.

Wrong approach:

def bad_metric(y_true, y_pred):
count = 0
for i in range(len(y_true)):
if y_true[i] == round(y_pred[i]):
count += 1
return count / len(y_true)

Correct approach:

def good_metric(y_true, y_pred):
y_pred_rounded = tf.round(y_pred)
correct = tf.cast(tf.equal(y_true, y_pred_rounded), tf.float32)
return tf.reduce_mean(correct)

Root cause:

Using Python loops and variables breaks TensorFlow's graph and batch processing, causing errors or slowdowns.

#2 Not resetting metric states between epochs in a stateful metric class.

Wrong approach:

class MyMetric(tf.keras.metrics.Metric):
def __init__(self):
super().__init__()
self.total = self.add_weight('total', initializer='zeros')
def update_state(self, y_true, y_pred):
self.total.assign_add(tf.reduce_sum(y_true))
def result(self):
return self.total

Correct approach:

class MyMetric(tf.keras.metrics.Metric):
def __init__(self):
super().__init__()
self.total = self.add_weight('total', initializer='zeros')
def update_state(self, y_true, y_pred):
self.total.assign_add(tf.reduce_sum(y_true))
def result(self):
return self.total
def reset_states(self):
self.total.assign(0)

Root cause:

Forgetting to reset states causes metric values to accumulate incorrectly across epochs.

#3 Using custom metrics that perform heavy computations or create new variables every batch.

Wrong approach:

class SlowMetric(tf.keras.metrics.Metric):
def update_state(self, y_true, y_pred):
for i in range(10000):
pass # heavy loop
self.total = self.add_weight('total', initializer='zeros')

Correct approach:

class FastMetric(tf.keras.metrics.Metric):
def __init__(self):
super().__init__()
self.total = self.add_weight('total', initializer='zeros')
def update_state(self, y_true, y_pred):
# vectorized operations only
pass

Root cause:

Heavy or repeated operations slow training and creating variables inside update_state breaks TensorFlow's variable management.

Key Takeaways

Custom metrics let you measure exactly what matters for your model beyond standard metrics.

You can write custom metrics as simple functions or as classes for more complex stateful tracking.

Proper handling of metric state and distribution is essential for accurate and reliable evaluation.

Efficient custom metrics keep training fast and scalable by using TensorFlow operations and avoiding Python loops.

Understanding custom metrics deepens your control over model evaluation and helps build better machine learning solutions.

Practice

1. What is the main purpose of a custom metric in TensorFlow model training?

easy

A. To track a special score based on true labels and predictions during training

B. To change the model's architecture automatically

C. To speed up the training process by using GPU

D. To save the model weights after each epoch

Solution

1. Step 1: Understand what metrics do

   Metrics measure how well the model performs by comparing true labels and predictions.

2. Step 2: Identify the role of custom metrics

   Custom metrics let you define your own way to measure performance during training.

3. Final Answer:

   To track a special score based on true labels and predictions during training -> Option A

4. Quick Check:

   Custom metric = special score tracking [OK]

Hint: Custom metrics measure model performance uniquely during training [OK]

Common Mistakes:

• Confusing metrics with model architecture changes
• Thinking metrics speed up training
• Believing metrics save model weights

2. Which of the following is the correct way to define a simple custom metric function in TensorFlow?

easy

A. def my_metric(y_true, y_pred): print(y_true, y_pred)

B. def my_metric(y_true, y_pred): return np.mean(np.abs(y_true - y_pred))

C. def my_metric(y_true, y_pred): return y_true + y_pred

D. def my_metric(y_true, y_pred): return tf.reduce_mean(tf.abs(y_true - y_pred))

Solution

1. Step 1: Check TensorFlow compatibility

   Custom metrics must use TensorFlow operations like tf.reduce_mean for compatibility.

2. Step 2: Identify correct function

   def my_metric(y_true, y_pred): return tf.reduce_mean(tf.abs(y_true - y_pred)) uses tf.reduce_mean and tf.abs correctly; def my_metric(y_true, y_pred): return np.mean(np.abs(y_true - y_pred)) uses numpy which is incompatible.

3. Final Answer:

   def my_metric(y_true, y_pred): return tf.reduce_mean(tf.abs(y_true - y_pred)) -> Option D

4. Quick Check:

   Use TensorFlow ops in custom metrics [OK]

Hint: Use TensorFlow ops, not numpy, inside custom metric functions [OK]

Common Mistakes:

• Using numpy functions inside metric causing errors
• Returning tensors without reduction to scalar
• Printing values instead of returning metric

3. Given the following custom metric code, what will be the output metric value for y_true = [1, 2, 3] and y_pred = [1, 2, 4]?

def custom_metric(y_true, y_pred):
    diff = tf.abs(y_true - y_pred)
    return tf.reduce_mean(diff)

medium

A. 1.0

B. 0.6667

C. 0.3333

D. 2.0

Solution

1. Step 1: Calculate absolute differences

   Differences: |1-1|=0, |2-2|=0, |3-4|=1

2. Step 2: Compute mean of differences

   Mean = (0 + 0 + 1) / 3 = 0.3333

3. Final Answer:

   0.3333 -> Option C

4. Quick Check:

   Mean absolute difference = 0.3333 [OK]

Hint: Calculate absolute differences, then average them [OK]

Common Mistakes:

• Summing differences without averaging
• Using squared difference instead of absolute
• Mixing up y_true and y_pred order

4. Identify the error in this custom metric function and choose the correct fix:

def faulty_metric(y_true, y_pred):
    return tf.reduce_sum(y_true - y_pred)

medium

A. Convert y_true and y_pred to numpy arrays first

B. Use tf.reduce_mean instead of tf.reduce_sum to get average error

C. Replace subtraction with addition

D. Add print statements to debug values

Solution

1. Step 1: Understand metric output

   Metrics should return a scalar representing average error, not total sum.

2. Step 2: Fix by averaging

   Replacing tf.reduce_sum with tf.reduce_mean returns average difference, a better metric.

3. Final Answer:

   Use tf.reduce_mean instead of tf.reduce_sum to get average error -> Option B

4. Quick Check:

   Average error metric needs reduce_mean [OK]

Hint: Use reduce_mean for average, not reduce_sum [OK]

Common Mistakes:

• Using reduce_sum causing large values
• Mixing TensorFlow and numpy operations
• Not returning a scalar metric

5. You want to create a custom metric that counts how many predictions differ from true labels by more than 0.5. Which of the following implementations is correct?

hard

A. def count_large_diff(y_true, y_pred):
diff = tf.abs(y_true - y_pred)
return tf.reduce_sum(tf.cast(diff > 0.5, tf.float32))

B. def count_large_diff(y_true, y_pred):
diff = y_true - y_pred
return tf.reduce_mean(diff > 0.5)

C. def count_large_diff(y_true, y_pred):
diff = tf.abs(y_true - y_pred)
return tf.reduce_sum(diff > 0.5)

D. def count_large_diff(y_true, y_pred):
diff = tf.abs(y_true - y_pred)
return tf.reduce_sum(tf.int32(diff > 0.5))

Solution

1. Step 1: Identify correct comparison and casting

   We must compare differences with 0.5 and cast boolean to float for summation.

2. Step 2: Check TensorFlow functions used

   def count_large_diff(y_true, y_pred): diff = tf.abs(y_true - y_pred) return tf.reduce_sum(tf.cast(diff > 0.5, tf.float32)) uses tf.abs, comparison, tf.cast to float32, and tf.reduce_sum correctly.

3. Final Answer:

   def count_large_diff(y_true, y_pred): diff = tf.abs(y_true - y_pred) return tf.reduce_sum(tf.cast(diff > 0.5, tf.float32)) -> Option A

4. Quick Check:

   Cast booleans to float before summing [OK]

Hint: Cast boolean mask to float before summing [OK]

Common Mistakes:

• Not casting boolean mask before summing
• Using reduce_mean instead of reduce_sum
• Using wrong data type cast like tf.int32