Custom loss functions in TensorFlow - Model Metrics & Evaluation

Metrics & Evaluation - Custom loss functions

Which metric matters for Custom Loss Functions and WHY

When you create a custom loss function, the main metric to watch is the loss value itself. This loss tells the model how well it is doing according to your special rule. The goal is to make this loss as small as possible during training.

Besides loss, you should also track accuracy or other task-specific metrics (like precision or recall) to see if your custom loss helps the model learn what you want.

Why? Because the loss guides the model's learning. If the loss is not meaningful, the model won't improve in the way you expect.

Confusion Matrix Example

For classification tasks using a custom loss, you can still use a confusion matrix to understand predictions:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP)  | False Positive (FP) |
      | False Negative (FN) | True Negative (TN)  |
    

These counts help calculate precision, recall, and F1 score, which show how well the model performs beyond just the loss.

Precision vs Recall Tradeoff with Custom Loss

Custom loss functions can be designed to focus on either precision or recall, depending on what matters more.

For example, if you want to catch all positive cases (like detecting fraud), you might design a loss that penalizes missing positives more, improving recall.

On the other hand, if you want to avoid false alarms (like spam filters), your loss might focus on precision.

Understanding this tradeoff helps you decide how to shape your custom loss.

Good vs Bad Metric Values for Custom Loss Functions

Good:

• Loss steadily decreases during training.
• Accuracy or task metrics improve alongside loss.
• Precision and recall values align with your goal (high recall if catching positives is key, high precision if avoiding false alarms is key).

Bad:

• Loss stays high or fluctuates wildly.
• Accuracy or other metrics do not improve or get worse.
• Precision and recall are both low, meaning the model is confused.

Common Pitfalls with Custom Loss Functions

• Incorrect loss math: A small mistake in the formula can make the loss meaningless.
• Ignoring scale: If the loss values are too large or too small, training can be unstable.
• Overfitting: Custom loss might cause the model to memorize training data if too strict.
• Data leakage: Using future or test data in loss calculation breaks learning.
• Not monitoring other metrics: Only watching loss can hide poor real-world performance.

Self Check: Your model has 98% accuracy but 12% recall on fraud. Is it good?

No, this model is not good for fraud detection. Even though accuracy is high, the recall is very low, meaning it misses most fraud cases. For fraud detection, catching fraud (high recall) is more important than overall accuracy.

Key Result

Custom loss functions must produce meaningful loss values that decrease during training and align with task metrics like precision and recall to ensure effective learning.

Practice

1. What is the main purpose of a custom loss function in TensorFlow?

easy

A. To change the model's architecture layers

B. To generate new training data from existing data

C. To speed up the training process automatically

D. To define a special way to measure how wrong the model's predictions are

Solution

1. Step 1: Understand the role of loss functions

   Loss functions measure how far the model's predictions are from the true values.

2. Step 2: Identify what custom loss functions do

   Custom loss functions let you define your own way to measure this error, which can help in special cases.

3. Final Answer:

   To define a special way to measure how wrong the model's predictions are -> Option D

4. Quick Check:

   Custom loss = custom error measure [OK]

Hint: Loss functions measure error; custom ones define your own error [OK]

Common Mistakes:

• Confusing loss function with model architecture
• Thinking loss speeds up training automatically
• Mixing loss function with data generation

2. Which of the following is the correct way to define a simple custom loss function in TensorFlow using Python?

easy

A. def custom_loss(y_true, y_pred): return tf.add(y_true, y_pred)

B. def custom_loss(y_true, y_pred): return y_true + y_pred

C. def custom_loss(y_true, y_pred): return tf.reduce_mean(tf.square(y_true - y_pred))

D. def custom_loss(y_true, y_pred): return tf.matmul(y_true, y_pred)

Solution

1. Step 1: Check the function signature

   The function must take true and predicted values as inputs.

2. Step 2: Verify the loss calculation

   Loss should measure error, e.g., mean squared error: mean of squared differences.

3. Final Answer:

   def custom_loss(y_true, y_pred): return tf.reduce_mean(tf.square(y_true - y_pred)) -> Option C

4. Quick Check:

   Loss = mean squared error formula [OK]

Hint: Loss = mean of squared differences for regression [OK]

Common Mistakes:

• Returning sum instead of mean
• Adding true and predicted values instead of difference
• Using matrix multiplication instead of error calculation

3. Given the following custom loss function, what will be the output if y_true = [2.0, 3.0] and y_pred = [1.0, 5.0]?

def custom_loss(y_true, y_pred):
    return tf.reduce_sum(tf.abs(y_true - y_pred))

medium

A. 3.0

B. 4.0

C. 5.0

D. 6.0

Solution

1. Step 1: Calculate absolute differences

   abs(2.0 - 1.0) = 1.0 and abs(3.0 - 5.0) = 2.0

2. Step 2: Sum the absolute differences

   1.0 + 2.0 = 3.0

3. Final Answer:

   3.0 -> Option A

4. Quick Check:

   Sum of absolute errors = 3.0 [OK]

Hint: Sum absolute differences for this loss [OK]

Common Mistakes:

• Confusing sum with mean
• Using squared differences instead of absolute
• Mixing up true and predicted values

4. Identify the error in this custom loss function and select the correct fix:

def custom_loss(y_true, y_pred):
    return tf.reduce_mean(y_true - y_pred)

medium

A. Change function to return y_true + y_pred

B. Use absolute difference or squared difference instead of direct subtraction

C. Remove tf.reduce_mean and return raw difference

D. Use tf.matmul instead of subtraction

Solution

1. Step 1: Understand the problem with direct subtraction

   Direct subtraction can give positive and negative values that cancel out, not measuring error properly.

2. Step 2: Correct the loss calculation

   Use absolute value or square the difference to measure error magnitude correctly.

3. Final Answer:

   Use absolute difference or squared difference instead of direct subtraction -> Option B

4. Quick Check:

   Loss must measure error magnitude, not signed difference [OK]

Hint: Loss must be positive; use abs() or square() [OK]

Common Mistakes:

• Returning signed difference causing cancellation
• Removing mean reduces loss meaning
• Using matrix multiplication wrongly

5. You want to create a custom loss function that penalizes under-predictions more than over-predictions. Which of the following custom loss functions achieves this behavior correctly?

hard

A. def custom_loss(y_true, y_pred): return tf.reduce_mean(tf.where(y_pred < y_true, 2 * (y_true - y_pred), y_pred - y_true))

B. def custom_loss(y_true, y_pred): return tf.reduce_mean(tf.abs(y_true - y_pred))

C. def custom_loss(y_true, y_pred): return tf.reduce_mean(tf.square(y_true - y_pred))

D. def custom_loss(y_true, y_pred): return tf.reduce_sum(y_pred - y_true)

Solution

1. Step 1: Understand the requirement

   We want to penalize under-predictions (y_pred < y_true) more than over-predictions.

2. Step 2: Analyze each option

   def custom_loss(y_true, y_pred): return tf.reduce_mean(tf.where(y_pred < y_true, 2 * (y_true - y_pred), y_pred - y_true)) uses tf.where to multiply the error by 2 when y_pred < y_true, increasing penalty for under-predictions. Options B and C treat errors symmetrically. def custom_loss(y_true, y_pred): return tf.reduce_sum(y_pred - y_true) sums signed differences, which is incorrect.

3. Final Answer:

   def custom_loss(y_true, y_pred): return tf.reduce_mean(tf.where(y_pred < y_true, 2 * (y_true - y_pred), y_pred - y_true)) -> Option A

4. Quick Check:

   Conditional penalty for under-prediction = def custom_loss(y_true, y_pred): return tf.reduce_mean(tf.where(y_pred < y_true, 2 * (y_true - y_pred), y_pred - y_true)) [OK]

Hint: Use tf.where to weight errors differently [OK]

Common Mistakes:

• Using symmetric loss ignoring under/over difference
• Summing signed differences causing wrong loss
• Not using conditional logic for penalty