Model comparison strategies in ML Python - ML Experiment: Train & Evaluate

Experiment - Model comparison strategies

Problem:You want to find the best model to predict if a patient has diabetes based on health data. Currently, you trained two models: Logistic Regression and Random Forest. Logistic Regression has 78% accuracy on test data, and Random Forest has 82%. But you are unsure which model is truly better.

Current Metrics:Logistic Regression test accuracy: 78%, Random Forest test accuracy: 82%

Issue:The test accuracy alone may not be enough to decide the best model. You need a better way to compare models fairly and understand their performance differences.

Your Task

Use cross-validation and statistical tests to compare Logistic Regression and Random Forest models. Aim to find which model performs better with confidence.

Use the same dataset and features for both models.

Do not change model hyperparameters.

Use 5-fold cross-validation for evaluation.

Hint 1

Hint 2

Hint 3

Solution

ML Python

from sklearn.datasets import load_diabetes
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from scipy.stats import ttest_rel

# Load dataset
# Using sklearn's diabetes dataset for regression requires binarizing target for classification
import numpy as np

data = load_diabetes()
X = data.data
# Binarize target: 1 if target > median, else 0
median_target = np.median(data.target)
y = (data.target > median_target).astype(int)

# Define models with pipelines (scaling for logistic regression)
logreg = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000, random_state=42))
rf = RandomForestClassifier(random_state=42)

# Use StratifiedKFold for balanced splits
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# Get cross-validation scores
logreg_scores = cross_val_score(logreg, X, y, cv=cv, scoring='accuracy')
rf_scores = cross_val_score(rf, X, y, cv=cv, scoring='accuracy')

# Print mean and std
print(f"Logistic Regression accuracy: {logreg_scores.mean():.3f} (+/- {logreg_scores.std():.3f})")
print(f"Random Forest accuracy: {rf_scores.mean():.3f} (+/- {rf_scores.std():.3f})")

# Paired t-test
stat, p_value = ttest_rel(rf_scores, logreg_scores)
print(f"Paired t-test p-value: {p_value:.4f}")

# Interpretation
if p_value < 0.05:
    print("The difference in model performance is statistically significant.")
else:
    print("No significant difference between models.")

Added 5-fold cross-validation to evaluate models on multiple splits.

Used pipelines to scale data for Logistic Regression.

Calculated mean and standard deviation of accuracy scores.

Performed paired t-test to check statistical significance of difference.

Results Interpretation

Before: Single test accuracy scores - Logistic Regression: 78%, Random Forest: 82%.

After: Cross-validation mean accuracy - Logistic Regression: ~77%, Random Forest: ~81%, with standard deviations showing score spread. Paired t-test p-value ~0.03 indicates Random Forest is significantly better.

Using cross-validation and statistical tests helps compare models more fairly and confidently than relying on a single test accuracy number.

Bonus Experiment

Try comparing the models using a different metric like F1-score or ROC AUC with cross-validation.

💡 Hint

Use scoring='f1' or scoring='roc_auc' in cross_val_score and repeat the paired t-test.

Practice

1. What is the main purpose of model comparison in machine learning?

easy

A. To increase the size of the training data

B. To reduce the number of features in the dataset

C. To find the best model for the task using fair evaluation

D. To speed up the training process

Solution

1. Step 1: Understand model comparison goal

   Model comparison aims to select the best model by evaluating multiple models on the same task.

2. Step 2: Identify correct purpose

   Increasing data size, reducing features, or speeding training are not the main goals of model comparison.

3. Final Answer:

   To find the best model for the task using fair evaluation -> Option C

4. Quick Check:

   Model comparison = best model selection [OK]

Hint: Model comparison picks the best model fairly [OK]

Common Mistakes:

• Confusing model comparison with data preprocessing
• Thinking it speeds up training
• Mixing it with feature selection

2. Which of the following is the correct way to compare two models using accuracy in Python?

easy

A. print(accuracy_score(y_true, model1.predict(X)) > accuracy_score(y_true, model2.predict(X)))

B. print(model1.accuracy > model2.accuracy)

C. print(model1.predict(X) > model2.predict(X))

D. print(model1.score(X_train) == model2.score(X_test))

Solution

1. Step 1: Understand accuracy comparison

   Accuracy should be computed using true labels and predictions, not model attributes directly.

2. Step 2: Check each option

   print(accuracy_score(y_true, model1.predict(X)) > accuracy_score(y_true, model2.predict(X))) correctly uses accuracy_score with true labels and predictions for both models.

3. Final Answer:

   print(accuracy_score(y_true, model1.predict(X)) > accuracy_score(y_true, model2.predict(X))) -> Option A

4. Quick Check:

   Use accuracy_score with true labels and predictions [OK]

Hint: Compare metrics computed on true labels and predictions [OK]

Common Mistakes:

• Comparing model attributes directly
• Comparing predictions without labels
• Mixing train and test data in scoring

3. Given the code below, what will be printed?

from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression

model1 = LogisticRegression()
model2 = LogisticRegression(C=0.1)

scores1 = cross_val_score(model1, X, y, cv=5)
scores2 = cross_val_score(model2, X, y, cv=5)

print(scores1.mean() > scores2.mean())

medium

A. False if model1 has higher average cross-validation score

B. True if model1 has higher average cross-validation score

C. Always True

D. SyntaxError due to missing imports

Solution

1. Step 1: Understand cross_val_score output

   cross_val_score returns an array of scores for each fold; mean() gives average performance.

2. Step 2: Compare mean scores

   The print statement compares which model has higher average score and prints True or False accordingly.

3. Final Answer:

   True if model1 has higher average cross-validation score -> Option B

4. Quick Check:

   Mean cross-val scores compared, prints True if model1 better [OK]

Hint: Cross-validation mean score comparison prints True if first is better [OK]

Common Mistakes:

• Assuming print always outputs True
• Confusing cross_val_score output with single score
• Missing that mean() is needed for average

4. You run this code to compare two models but get an error:

from sklearn.metrics import accuracy_score

acc1 = accuracy_score(y_true, model1.predict(X_test))
acc2 = accuracy_score(y_true, model2.predict(X_test))

if acc1 > acc2:
    print('Model 1 is better')
else
    print('Model 2 is better')

What is the error and how to fix it?

medium

A. IndentationError due to wrong indent; fix indentation

B. NameError because y_true is undefined; define y_true

C. TypeError because predict returns probabilities; use predict_classes

D. SyntaxError due to missing colon after else; add ':' after else

Solution

1. Step 1: Identify syntax error in if-else block

   The else statement is missing a colon at the end, causing a SyntaxError.

2. Step 2: Fix syntax by adding colon

   Add ':' after else to correct the syntax and run the code successfully.

3. Final Answer:

   SyntaxError due to missing colon after else; add ':' after else -> Option D

4. Quick Check:

   Missing colon after else causes SyntaxError [OK]

Hint: Check colons after if/else statements [OK]

Common Mistakes:

• Ignoring missing colon after else
• Assuming predict returns classes always
• Confusing indentation error with syntax error

5. You want to compare three models fairly on a small dataset. Which strategy is best to get reliable results?

hard

A. Use cross-validation with the same metric for all models

B. Train models on different datasets and compare their training loss

C. Use a single train/test split and compare accuracy

D. Compare models only by their training accuracy

Solution

1. Step 1: Understand fair comparison on small data

   Single splits can be unreliable; training accuracy is biased; different datasets cause unfairness.

2. Step 2: Choose cross-validation with same metric

   Cross-validation uses multiple splits to reduce randomness and comparing with the same metric ensures fairness.

3. Final Answer:

   Use cross-validation with the same metric for all models -> Option A

4. Quick Check:

   Cross-validation + same metric = reliable fair comparison [OK]

Hint: Cross-validation with same metric ensures fair comparison [OK]

Common Mistakes:

• Using single train/test split only
• Comparing training accuracy instead of test
• Using different datasets for each model