ROC and AUC curves in TensorFlow - ML Experiment: Train & Evaluate

Experiment - ROC and AUC curves

Problem:You have trained a binary classification model using TensorFlow. The model achieves 95% training accuracy but only 75% validation accuracy. You want to evaluate the model's ability to distinguish between classes using ROC and AUC metrics.

Current Metrics:Training accuracy: 95%, Validation accuracy: 75%, No ROC or AUC metrics computed yet.

Issue:The model shows signs of overfitting and the current accuracy metrics do not fully capture the model's discrimination power. ROC and AUC curves are needed to better understand performance.

Your Task

Calculate and plot the ROC curve and compute the AUC score for the validation data to evaluate the model's classification performance beyond accuracy.

Use TensorFlow and related libraries only.

Do not change the model architecture or training process.

Use the existing validation dataset for evaluation.

Hint 1

Hint 2

Hint 3

Solution

TensorFlow

import tensorflow as tf
from sklearn.metrics import roc_curve, auc
import matplotlib.pyplot as plt
import numpy as np

# Assume X_val and y_val are validation features and labels
# Assume model is already trained

# Get predicted probabilities for the positive class
y_pred_prob = model.predict(X_val).ravel()

# Compute ROC curve and ROC area
fpr, tpr, thresholds = roc_curve(y_val, y_pred_prob)
roc_auc = auc(fpr, tpr)

# Plot ROC curve
plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, color='darkorange', lw=2, label=f'ROC curve (area = {roc_auc:.2f})')
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic (ROC) Curve')
plt.legend(loc='lower right')
plt.grid(True)
plt.show()

print(f'AUC score: {roc_auc:.4f}')

Added code to predict probabilities on validation data.

Computed false positive rate, true positive rate, and thresholds using sklearn.metrics.roc_curve.

Calculated AUC score using sklearn.metrics.auc.

Plotted ROC curve with matplotlib to visualize model performance.

Results Interpretation

Before: Only accuracy metrics were available: Training accuracy 95%, Validation accuracy 75%. These metrics do not show how well the model distinguishes classes.

After: ROC curve plotted and AUC score computed as 0.82. This shows the model has good discrimination ability despite lower validation accuracy.

ROC and AUC provide a better understanding of a binary classifier's performance by measuring its ability to separate classes across all classification thresholds, which accuracy alone cannot reveal.

Bonus Experiment

Try adding dropout layers to the model and retrain. Then compute ROC and AUC again to see if overfitting reduces and validation AUC improves.

💡 Hint

Dropout randomly disables neurons during training, which helps the model generalize better and can improve validation metrics like AUC.