Why proper evaluation prevents overfitting in ML Python - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why do we use a separate test set in machine learning?

Imagine you train a model on some data and then check how well it works on the same data. Why is it important to use a different set of data (test set) to evaluate the model?

A Because the test set helps us see how the model performs on new, unseen data, preventing us from thinking it works better than it really does.

B Because the test set is used to train the model faster by giving it extra examples.

C Because the test set contains only easy examples that make the model look good.

D Because the test set is used to tune the model’s parameters during training.

💡 Hint

Think about how practicing a skill on the same questions might not prepare you for new questions.

Using a separate test set shows how the model will perform on data it has never seen before. This helps detect if the model is just memorizing the training data (overfitting) rather than learning general patterns.

❓ Predict Output

intermediate

What is the output of this model evaluation code?

Given the following code that splits data and evaluates a model, what will be printed?

ML Python

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
model = DecisionTreeClassifier(random_state=42)
model.fit(X_train, y_train)
predictions_train = model.predict(X_train)
predictions_test = model.predict(X_test)
print(f"Train accuracy: {accuracy_score(y_train, predictions_train):.2f}")
print(f"Test accuracy: {accuracy_score(y_test, predictions_test):.2f}")

A

Train accuracy: 1.00
Test accuracy: 0.98

B

Train accuracy: 0.98
Test accuracy: 1.00

C

Train accuracy: 0.50
Test accuracy: 0.50

D

Train accuracy: 0.85
Test accuracy: 0.85

💡 Hint

Decision trees can perfectly fit training data but may perform slightly worse on test data.

The decision tree perfectly fits the training data (accuracy 1.00) but has slightly lower accuracy on test data (around 0.98), showing some generalization.

❓ Metrics

advanced

Which metric best detects overfitting in classification?

You train a classifier and get very high accuracy on training data but much lower accuracy on test data. Which metric helps you best understand this overfitting problem?

A Recall on test data only

B Precision on training data only

C F1 score on training data only

D Difference between training accuracy and test accuracy

💡 Hint

Think about comparing performance on training vs test data.

The difference between training and test accuracy shows if the model performs much better on training data than on new data, indicating overfitting.

❓ Model Choice

advanced

Which model choice helps reduce overfitting?

You want to prevent overfitting on a small dataset. Which model choice is best?

A Use the training data as the test data

B Use a simpler model with fewer parameters

C Use no validation and train longer

D Use a very deep neural network with many layers

💡 Hint

Simpler models tend to generalize better on small data.

Simpler models are less likely to memorize training data and more likely to find general patterns, reducing overfitting.

🔧 Debug

expert

Why does this evaluation code give misleading results?

Look at this code snippet. It trains and evaluates a model but the evaluation results are misleading. Why?

ML Python

from sklearn.datasets import load_digits
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

data = load_digits()
X, y = data.data, data.target
model = DecisionTreeClassifier()
model.fit(X, y)
predictions = model.predict(X)
print(f"Accuracy: {accuracy_score(y, predictions):.2f}")

A Because accuracy_score requires test data, not training data.

B Because the model was not trained at all before prediction.

C Because the model is evaluated on the same data it was trained on, causing overfitting to be hidden.

D Because the dataset is too small to train any model.

💡 Hint

Think about what happens if you test on the same data you trained on.

Evaluating on training data shows overly optimistic accuracy because the model memorized the data. Proper evaluation requires separate test data.

Practice

1. Why is it important to evaluate a machine learning model on data it has never seen before?

easy

A. To check if the model can make good predictions on new data

B. To make the training process faster

C. To increase the size of the training data

D. To reduce the number of features in the data

Solution

1. Step 1: Understand model evaluation purpose

   Evaluating on unseen data tests if the model generalizes beyond training examples.

2. Step 2: Identify the goal of evaluation

   The goal is to ensure the model predicts well on new, real-world data, not just memorized training data.

3. Final Answer:

   To check if the model can make good predictions on new data -> Option A

4. Quick Check:

   Evaluation on new data = Check generalization [OK]

Hint: Test on new data to check real-world performance [OK]

Common Mistakes:

• Thinking evaluation speeds up training
• Confusing evaluation with feature selection
• Believing evaluation increases training data size

2. Which of the following is the correct way to split data for proper model evaluation?

easy

A. Split data into training and testing sets

B. Use only testing data for training

C. Use all data for training and testing

D. Train on test data and evaluate on training data

Solution

1. Step 1: Recall data splitting best practice

   Proper evaluation requires separating data into training and testing sets to avoid bias.

2. Step 2: Identify correct splitting method

   Splitting data into training and testing sets ensures the model is evaluated on unseen data.

3. Final Answer:

   Split data into training and testing sets -> Option A

4. Quick Check:

   Data split = Training + Testing [OK]

Hint: Always keep test data separate from training data [OK]

Common Mistakes:

• Training and testing on the same data
• Using test data for training
• Not splitting data at all

3. Consider this Python code snippet for evaluating a model:

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

X = [[1], [2], [3], [4], [5], [6]]
y = [0, 0, 0, 1, 1, 1]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
model = LogisticRegression()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(round(accuracy, 2))

What will be the printed accuracy?

medium

A. 0.0

B. 0.67

C. 0.5

D. 1.0

Solution

1. Step 1: Understand data split and model training

   Data is split with 33% test size, so test set has 2 samples. Model trains on 4 samples.

2. Step 2: Predict and calculate accuracy

   With logistic regression on this split and data, the model predicts 1 out of 2 test samples correctly, giving accuracy 0.5.

3. Final Answer:

   0.5 -> Option C

4. Quick Check:

   Correct predictions / total test samples = 0.5 [OK]

Hint: Check test set size and prediction correctness [OK]

Common Mistakes:

• Assuming accuracy is always less than 1
• Ignoring test set size
• Confusing train and test data

4. You trained a model and got 100% accuracy on training data but only 50% on test data. What is the most likely problem?

medium

A. The training data is too small

B. The model is underfitting the training data

C. The test data is too easy

D. The model is overfitting the training data

Solution

1. Step 1: Analyze accuracy difference

   Perfect training accuracy but low test accuracy suggests the model memorized training data but fails on new data.

2. Step 2: Identify overfitting

   This behavior is classic overfitting, where the model fits training data too closely and does not generalize.

3. Final Answer:

   The model is overfitting the training data -> Option D

4. Quick Check:

   High train accuracy + low test accuracy = Overfitting [OK]

Hint: Big train-test accuracy gap means overfitting [OK]

Common Mistakes:

• Confusing overfitting with underfitting
• Blaming test data difficulty
• Ignoring accuracy differences

5. You have a dataset with 1000 samples. You want to prevent overfitting and get a reliable estimate of model performance. Which evaluation strategy is best?

hard

A. Split data once into 90% train and 10% test

B. Use 5-fold cross-validation to train and evaluate

C. Train on all data and report training accuracy

D. Evaluate only on training data with early stopping

Solution

1. Step 1: Understand evaluation strategies

   Training on all data or evaluating only on training data risks overfitting and unreliable performance estimates.

2. Step 2: Compare splitting vs cross-validation

   Single split may give biased results depending on the split. 5-fold cross-validation averages results over 5 splits, reducing bias and variance.

3. Step 3: Choose best method for reliable evaluation

   5-fold cross-validation is best to prevent overfitting and get trustworthy performance estimates on limited data.

4. Final Answer:

   Use 5-fold cross-validation to train and evaluate -> Option B

5. Quick Check:

   Cross-validation = Reliable evaluation [OK]

Hint: Use cross-validation for reliable performance estimates [OK]

Common Mistakes:

• Using training accuracy as performance
• Relying on a single train-test split
• Ignoring overfitting risks