Fine-tuning approach in TensorFlow - Model Metrics & Evaluation

Metrics & Evaluation - Fine-tuning approach

Which metric matters for Fine-tuning and WHY

Fine-tuning means adjusting a pre-trained model to a new task. The best metric depends on the task type:

• Classification: Accuracy, Precision, Recall, and F1-score matter to check if the model correctly labels new data.
• Regression: Mean Squared Error (MSE) or Mean Absolute Error (MAE) show how close predictions are to real values.

Because fine-tuning often uses small new data, overfitting is a risk. So, monitoring validation loss and validation accuracy helps know if the model learns well without memorizing.

Confusion Matrix Example for Fine-tuned Classification

Imagine a fine-tuned model classifying cats vs dogs. Here is a confusion matrix after testing 100 images:

      | Predicted Cat | Predicted Dog |
      |---------------|---------------|
      | True Cat: 40  | False Dog: 5  |
      | False Cat: 5  | True Dog: 50  |
    

From this:

• True Positives (TP) = 40 (cats correctly identified)
• False Positives (FP) = 5 (dogs wrongly called cats)
• True Negatives (TN) = 50 (dogs correctly identified)
• False Negatives (FN) = 5 (cats missed)

Precision vs Recall Tradeoff in Fine-tuning

Fine-tuning can change how the model balances precision and recall:

• Precision = TP / (TP + FP): How many predicted positives are correct?
• Recall = TP / (TP + FN): How many actual positives did we find?

Example: For a medical image model fine-tuned to detect disease:

• High recall is critical to catch all sick patients (avoid missing any).
• High precision avoids false alarms (not telling healthy people they are sick).

Fine-tuning can adjust this balance by changing training data or layers trained.

Good vs Bad Metric Values for Fine-tuning

Good fine-tuning results show:

• Validation accuracy close to or better than training accuracy (no big gap).
• Precision and recall both high (above 0.8) for balanced tasks.
• Loss decreasing steadily on training and validation sets.

Bad signs include:

• Validation accuracy much lower than training (overfitting).
• Precision very high but recall very low, or vice versa (unbalanced).
• Validation loss increasing while training loss decreases.

Common Metric Pitfalls in Fine-tuning

• Accuracy paradox: High accuracy can be misleading if classes are imbalanced.
• Data leakage: Using test data during fine-tuning inflates metrics falsely.
• Overfitting: Fine-tuning on small data can cause the model to memorize, not generalize.
• Ignoring validation metrics: Only looking at training metrics hides poor real-world performance.

Self-Check Question

Your fine-tuned model has 98% accuracy but only 12% recall on the positive class (e.g., fraud detection). Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses most positive cases (fraud). Even with high accuracy, it fails the main goal of catching fraud. You should improve recall before using it.

Key Result

Fine-tuning success depends on balanced metrics like precision and recall, and avoiding overfitting by monitoring validation performance.