Threshold tuning in ML Python - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is threshold tuning in machine learning?

Threshold tuning is the process of adjusting the cutoff value that decides how a model's prediction score is converted into a final class label, usually in classification tasks.

beginner

Why do we need to tune the threshold instead of always using 0.5?

Because the default threshold of 0.5 may not give the best balance between detecting positive cases and avoiding false alarms, especially when classes are imbalanced or costs of errors differ.

intermediate

How does changing the threshold affect precision and recall?

Increasing the threshold usually increases precision but lowers recall, while decreasing the threshold usually increases recall but lowers precision.

intermediate

What metric can help choose the best threshold for a binary classifier?

Metrics like F1 score, Youden's J statistic, or maximizing the Youden's J statistic on the ROC curve can help find the best threshold that balances true positives and false positives.

intermediate

Describe a simple method to find the optimal threshold using model predictions.

One simple method is to try many threshold values between 0 and 1, calculate the chosen metric (like F1 score) for each, and pick the threshold that gives the best metric value.

What does threshold tuning adjust in a classification model?

A The cutoff value to decide class labels from prediction scores

B The number of layers in the model

C The learning rate during training

D The size of the training dataset

Threshold tuning changes the cutoff value that converts prediction probabilities into class labels.

If you increase the threshold in a binary classifier, what usually happens to recall?

A Recall increases

B Recall decreases

C Recall stays the same

D Recall becomes zero

Increasing the threshold makes the model more strict, so it predicts fewer positives, lowering recall.

Which metric is commonly used to balance precision and recall when tuning thresholds?

A Mean squared error

B Accuracy

C F1 score

D Log loss

F1 score combines precision and recall into one metric, useful for threshold tuning.

Why might the default threshold of 0.5 not be ideal?

A Because it always maximizes accuracy

B Because it only works for regression

C Because it is too low for all models

D Because it ignores class imbalance and error costs

The 0.5 threshold does not consider if one class is rare or if some errors are more costly.

What is a simple way to find the best threshold?

A Try many thresholds and pick the one with the best metric

B Randomly pick a threshold

C Always use 0.5

D Use the threshold that gives the lowest loss during training

Testing multiple thresholds and selecting the best based on a metric is a common approach.

Explain what threshold tuning is and why it is important in classification models.

Think about how the model decides positive or negative predictions.

Describe how changing the threshold affects precision and recall, and how you might choose the best threshold.

Consider what happens when you make the model more or less strict.

Practice

1. What is the main purpose of threshold tuning in machine learning classification?

easy

A. To find the best cutoff probability to decide between classes

B. To increase the size of the training dataset

C. To reduce the number of features used in the model

D. To speed up the training process

Solution

1. Step 1: Understand threshold tuning concept

   Threshold tuning is about choosing a cutoff value for predicted probabilities to decide class labels.

2. Step 2: Identify the main goal

   The goal is to find the cutoff that best separates positive and negative classes for better decisions.

3. Final Answer:

   To find the best cutoff probability to decide between classes -> Option A

4. Quick Check:

   Threshold tuning = best cutoff choice [OK]

Hint: Threshold tuning picks the cutoff to decide yes/no [OK]

Common Mistakes:

• Confusing threshold tuning with feature selection
• Thinking threshold tuning changes training data size
• Assuming threshold tuning speeds up training

2. Which of the following is the correct way to apply a threshold of 0.7 to predicted probabilities probs in Python to get binary predictions?

easy

A. preds = (probs > 0.7).astype(int)

B. preds = probs > 0.7

C. preds = int(probs > 0.7)

D. preds = probs >= 0.7

Solution

1. Step 1: Understand threshold application

   We compare each probability to 0.7 to get True/False, then convert to 0/1 integers.

2. Step 2: Check correct syntax

   Using (probs > 0.7).astype(int) converts boolean array to integer array correctly.

3. Final Answer:

   preds = (probs > 0.7).astype(int) -> Option A

4. Quick Check:

   Threshold applied with boolean then int cast [OK]

Hint: Use boolean comparison then convert to int for binary labels [OK]

Common Mistakes:

• Forgetting to convert boolean to int
• Using int() on entire array instead of element-wise
• Using >= instead of > changes threshold logic

3. Given the following code, what will be the printed F1 score after threshold tuning?

from sklearn.metrics import f1_score
probs = [0.2, 0.8, 0.6, 0.4]
true_labels = [0, 1, 1, 0]
threshold = 0.5
preds = [1 if p > threshold else 0 for p in probs]
f1 = f1_score(true_labels, preds)
print(round(f1, 2))

medium

A. 0.80

B. 0.67

C. 1.00

D. 0.50

Solution

1. Step 1: Calculate predictions with threshold 0.5

   probs > 0.5 gives preds = [0, 1, 1, 0]

2. Step 2: Compute F1 score for preds vs true_labels

   True positives = 2, false positives = 0, false negatives = 0, so F1 = 2*TP/(2*TP+FP+FN) = 2*2/(4+0+0) = 1.0, since preds and true_labels are identical.

3. Final Answer:

   1.00 -> Option C

4. Quick Check:

   Perfect match means F1 = 1.00 [OK]

Hint: Check predicted labels carefully before scoring [OK]

Common Mistakes:

• Miscomputing predictions from threshold
• Confusing precision and recall in F1 calculation
• Rounding errors in final score

4. The following code tries to tune threshold but gives an error. What is the error?

probs = [0.1, 0.4, 0.6, 0.9]
true_labels = [0, 0, 1, 1]
thresholds = [0.3, 0.5, 0.7]
best_f1 = 0
for t in thresholds:
    preds = (probs > t)
    f1 = f1_score(true_labels, preds)
    if f1 > best_f1:
        best_f1 = f1
print(best_f1)

medium

A. Thresholds list is empty

B. Missing import of f1_score

C. preds is boolean, should be integers

D. Loop variable t is not used

Solution

1. Step 1: Check code for missing imports

   The code uses f1_score but does not import it from sklearn.metrics.

2. Step 2: Identify error cause

   Without importing f1_score, Python will raise a NameError when calling f1_score.

3. Final Answer:

   Missing import of f1_score -> Option B

4. Quick Check:

   Always import functions before use [OK]

Hint: Check if all functions are imported before use [OK]

Common Mistakes:

• Assuming boolean preds cause error (they don't)
• Ignoring missing import errors
• Thinking loop variable is unused

5. You have a model predicting probabilities for a rare disease. You want to tune the threshold to catch as many sick patients as possible but avoid too many false alarms. Which approach best balances this trade-off?

hard

A. Choose threshold maximizing recall only

B. Choose threshold minimizing accuracy

C. Choose threshold maximizing precision only

D. Choose threshold maximizing F1 score

Solution

1. Step 1: Understand the trade-off

   High recall catches more sick patients but may increase false alarms; precision reduces false alarms but may miss sick patients.

2. Step 2: Identify best metric for balance

   F1 score balances precision and recall, making it best to tune threshold for this trade-off.

3. Final Answer:

   Choose threshold maximizing F1 score -> Option D

4. Quick Check:

   F1 balances recall and precision [OK]

Hint: Use F1 score to balance recall and precision [OK]

Common Mistakes:

• Maximizing recall ignores false alarms
• Maximizing precision ignores missed cases
• Minimizing accuracy is not meaningful