ROC curve and AUC in ML Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding the ROC Curve

What does the ROC curve represent in a binary classification model?

A The trade-off between true positive rate and false positive rate at various thresholds

B The relationship between precision and recall for different classes

C The distribution of predicted probabilities for positive class only

D The accuracy of the model over different training epochs

💡 Hint

Think about what the axes of the ROC curve show.

The ROC curve plots the true positive rate (sensitivity) against the false positive rate (1 - specificity) at different classification thresholds, showing the trade-off between detecting positives and avoiding false alarms.

❓ Predict Output

intermediate

Calculating AUC from Model Predictions

Given the following true labels and predicted probabilities, what is the AUC score?

y_true = [0, 0, 1, 1]
y_scores = [0.1, 0.4, 0.35, 0.8]

ML Python

from sklearn.metrics import roc_auc_score

y_true = [0, 0, 1, 1]
y_scores = [0.1, 0.4, 0.35, 0.8]
auc = roc_auc_score(y_true, y_scores)
print(round(auc, 2))

A 0.95

B 0.85

C 0.65

D 0.75

💡 Hint

Use the sklearn roc_auc_score function and round to two decimals.

The AUC score here is 0.75, meaning the model has a 75% chance of ranking a random positive higher than a random negative.

❓ Metrics

advanced

Interpreting AUC Values

Which statement best describes an AUC value of 0.5 for a binary classifier?

A The model perfectly separates positive and negative classes

B The model performs no better than random guessing

C The model has high precision but low recall

D The model has a high false positive rate

💡 Hint

Think about what an AUC of 0.5 means in terms of ranking positives and negatives.

An AUC of 0.5 means the model's predictions are no better than flipping a coin, showing no ability to distinguish between classes.

🔧 Debug

advanced

Fixing Incorrect ROC Curve Plot

Consider this code snippet to plot an ROC curve. What is the main issue causing the plot to be incorrect?

from sklearn.metrics import roc_curve
import matplotlib.pyplot as plt

y_true = [0, 1, 0, 1]
y_scores = [0.2, 0.8, 0.4, 0.6]
fpr, tpr, thresholds = roc_curve(y_true, y_true)
plt.plot(fpr, tpr)
plt.show()

A The roc_curve function is called with true labels instead of predicted scores

B The matplotlib import is incorrect

C The y_true list contains invalid values

D The plot function is missing labels for axes

💡 Hint

Check the arguments passed to roc_curve.

The roc_curve function requires true labels and predicted scores. Passing true labels twice causes incorrect ROC data.

❓ Model Choice

expert

Choosing Model Based on AUC for Imbalanced Data

You have two binary classifiers evaluated on a highly imbalanced dataset. Model A has an AUC of 0.92, and Model B has an AUC of 0.85. However, Model B has higher precision on the minority class. Which model should you choose if your priority is to correctly identify positive cases while minimizing false alarms?

A Choose Model A because it has higher recall on the minority class

B Choose Model A because higher AUC means better overall ranking

C Choose Model B because higher precision reduces false alarms on positives

D Choose Model B because AUC is not useful for imbalanced data

💡 Hint

Consider what precision means in the context of false alarms.

When false alarms are costly, higher precision is more important than AUC. Model B's higher precision means fewer false positives, which suits the priority.

Practice

1. What does the ROC curve primarily show in a classification model?

easy

A. The trade-off between true positive rate and false positive rate at different thresholds

B. The accuracy of the model on the training data

C. The number of correct predictions made by the model

D. The loss value during model training

Solution

1. Step 1: Understand ROC curve components

   The ROC curve plots the true positive rate (sensitivity) against the false positive rate (1 - specificity) at various threshold settings.

2. Step 2: Identify what ROC shows

   This curve shows how well the model can distinguish between classes by varying the decision threshold.

3. Final Answer:

   The trade-off between true positive rate and false positive rate at different thresholds -> Option A

4. Quick Check:

   ROC curve = True Positive Rate vs False Positive Rate [OK]

Hint: ROC curve plots TPR vs FPR at thresholds [OK]

Common Mistakes:

• Confusing ROC with accuracy
• Thinking ROC shows loss values
• Assuming ROC shows only one threshold

2. Which of the following Python code snippets correctly imports the function to compute AUC from sklearn?

easy

A. import roc_auc_score from sklearn.metrics

B. from sklearn.metrics import auc_roc_score

C. from sklearn.metrics import roc_auc_score

D. import sklearn.metrics.roc_auc_score

Solution

1. Step 1: Recall correct import syntax

   In Python, functions are imported using 'from module import function' syntax.

2. Step 2: Identify correct function name

   The correct function name to compute AUC is 'roc_auc_score' from 'sklearn.metrics'.

3. Final Answer:

   from sklearn.metrics import roc_auc_score -> Option C

4. Quick Check:

   Correct import syntax = from sklearn.metrics import roc_auc_score [OK]

Hint: Use 'from sklearn.metrics import roc_auc_score' [OK]

Common Mistakes:

• Using wrong import syntax
• Misspelling function name
• Trying to import module as function

3. Given the following code, what is the printed AUC value?

from sklearn.metrics import roc_auc_score
true_labels = [0, 0, 1, 1]
predicted_scores = [0.1, 0.4, 0.35, 0.8]
auc = roc_auc_score(true_labels, predicted_scores)
print(round(auc, 2))

medium

A. 0.75

B. 0.85

C. 0.65

D. 0.95

Solution

1. Step 1: Understand roc_auc_score calculation

   The AUC measures how well the predicted scores rank positive samples higher than negative ones.

2. Step 2: Calculate AUC for given data

   Using the pairs, the AUC is 0.75 as the positive samples are ranked correctly most of the time but not perfectly.

3. Final Answer:

   0.75 -> Option A

4. Quick Check:

   AUC for given scores = 0.75 [OK]

Hint: Use roc_auc_score to get AUC directly [OK]

Common Mistakes:

• Confusing AUC with accuracy
• Rounding incorrectly
• Mixing up true labels and scores

4. Identify the error in the following code that tries to plot an ROC curve:

from sklearn.metrics import roc_curve
fpr, tpr, thresholds = roc_curve(y_true, y_pred)
plt.plot(fpr, tpr)
plt.show()

medium

A. roc_curve function is not imported correctly

B. y_pred should be predicted probabilities, not class labels

C. plt.plot should use thresholds instead of fpr

D. Missing import statement for numpy

Solution

1. Step 1: Check input requirements for roc_curve

   The roc_curve function requires true labels and predicted scores or probabilities, not class labels.

2. Step 2: Identify the error in inputs

   Using predicted class labels instead of probabilities causes incorrect ROC curve calculation.

3. Final Answer:

   y_pred should be predicted probabilities, not class labels -> Option B

4. Quick Check:

   Use probabilities for roc_curve inputs [OK]

Hint: Use predicted probabilities, not labels, for roc_curve [OK]

Common Mistakes:

• Passing class labels instead of probabilities
• Forgetting to import matplotlib.pyplot as plt
• Plotting thresholds instead of FPR and TPR

5. You have two models with these AUC scores: Model A = 0.82, Model B = 0.79. You want to improve Model B's performance. Which approach is best to increase the AUC score?

hard

A. Reduce the size of the training dataset to avoid overfitting

B. Tune the classification threshold to balance true positive and false positive rates

C. Use accuracy as the main metric instead of AUC

D. Increase the number of training epochs without changing data or model

Solution

1. Step 1: Understand what affects AUC

   AUC measures the model's ranking ability and is threshold-independent.

2. Step 2: Choose the best method to improve AUC

   Tuning the classification threshold does not improve AUC because AUC is threshold-independent; instead, improving the model's discriminative ability or training is needed.

3. Step 3: Evaluate options

   Increasing training epochs (Increase the number of training epochs without changing data or model) can help but may cause overfitting if not managed properly. Reducing dataset size (Reduce the size of the training dataset to avoid overfitting) usually harms performance. Using accuracy (Use accuracy as the main metric instead of AUC) does not improve AUC. Tuning threshold (Tune the classification threshold to balance true positive and false positive rates) affects operating point but not AUC.

4. Final Answer:

   Increase the number of training epochs without changing data or model -> Option D

5. Quick Check:

   More epochs can improve model performance and AUC [OK]

Hint: More training epochs improve AUC [OK]

Common Mistakes:

• Thinking threshold tuning affects AUC
• Reducing data size harms performance
• Switching to accuracy doesn't improve AUC