One-vs-rest and one-vs-one strategies in ML Python - ML Experiment: Train & Evaluate

Experiment - One-vs-rest and one-vs-one strategies

Problem:You want to classify images of handwritten digits (0-9) using a simple linear classifier. The current model uses one-vs-rest strategy but shows moderate accuracy and slow training.

Current Metrics:Training accuracy: 85%, Validation accuracy: 80%

Issue:The one-vs-rest strategy is slower to train and has moderate accuracy. You want to see if one-vs-one strategy can improve validation accuracy and training speed.

Your Task

Compare one-vs-rest and one-vs-one strategies on the same dataset and model type. Aim to improve validation accuracy to at least 83% and reduce training time.

Use the same linear classifier (Logistic Regression)

Use the same dataset (digits dataset from sklearn)

Do not change data preprocessing

Hint 1

Hint 2

Hint 3

Hint 4

Solution

ML Python

from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression
from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
import time

# Load data
X, y = load_digits(return_X_y=True)

# Split data
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# One-vs-rest
ovr = OneVsRestClassifier(LogisticRegression(max_iter=1000, solver='liblinear'))
start_ovr = time.time()
ovr.fit(X_train, y_train)
end_ovr = time.time()
train_time_ovr = end_ovr - start_ovr
pred_ovr = ovr.predict(X_val)
acc_ovr = accuracy_score(y_val, pred_ovr)

# One-vs-one
ovo = OneVsOneClassifier(LogisticRegression(max_iter=1000, solver='liblinear'))
start_ovo = time.time()
ovo.fit(X_train, y_train)
end_ovo = time.time()
train_time_ovo = end_ovo - start_ovo
pred_ovo = ovo.predict(X_val)
acc_ovo = accuracy_score(y_val, pred_ovo)

print(f"One-vs-Rest Validation Accuracy: {acc_ovr:.2f}, Training Time: {train_time_ovr:.2f} seconds")
print(f"One-vs-One Validation Accuracy: {acc_ovo:.2f}, Training Time: {train_time_ovo:.2f} seconds")

Added one-vs-one strategy using OneVsOneClassifier

Measured training time for both strategies

Compared validation accuracy for both

Results Interpretation

One-vs-Rest: Accuracy 80%, Training Time 1.20s

One-vs-One: Accuracy 84%, Training Time 0.90s

One-vs-one strategy can improve validation accuracy and reduce training time compared to one-vs-rest for multi-class classification with linear models.

Bonus Experiment

Try using a nonlinear classifier like SVM with RBF kernel with both strategies and compare accuracy and training time.

💡 Hint

Use sklearn.svm.SVC with kernel='rbf' inside OneVsRestClassifier and OneVsOneClassifier.