One-vs-rest and one-vs-one strategies in ML Python

Introduction

These strategies help us teach a computer to tell apart many groups by breaking the problem into simpler yes/no questions.

When you want to classify emails into categories like work, personal, or spam.

When sorting pictures of animals into types like cats, dogs, or birds.

When recognizing handwritten digits from 0 to 9.

When you have many classes but your model only knows how to separate two classes at a time.

When you want a simple way to extend a two-class model to many classes.

Syntax

ML Python

One-vs-rest:
Train one model per class, where that class is positive and all others are negative.

One-vs-one:
Train one model for every pair of classes, each distinguishing between just those two.

One-vs-rest creates as many models as classes.

One-vs-one creates models for every pair of classes, so number of models grows quickly.

Examples

Each model focuses on a simpler yes/no question.

ML Python

One-vs-rest example:
Classify if an email is 'spam' or 'not spam' ignoring other categories.

One-vs-one example:
Classify if an animal is 'cat' or 'dog' ignoring other animals.

One-vs-one uses more models but each is simpler.

ML Python

If you have 3 classes: A, B, C
One-vs-rest trains 3 models:
- A vs rest
- B vs rest
- C vs rest

One-vs-one trains 3 models:
- A vs B
- A vs C
- B vs C

Sample Model

This code trains two types of classifiers on the iris flower data and compares their accuracy.

ML Python

from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier
from sklearn.metrics import accuracy_score

# Load iris dataset (3 classes)
iris = datasets.load_iris()
X, y = iris.data, iris.target

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

# One-vs-rest classifier
ovr = OneVsRestClassifier(SVC(kernel='linear', probability=True))
ovr.fit(X_train, y_train)
y_pred_ovr = ovr.predict(X_test)
acc_ovr = accuracy_score(y_test, y_pred_ovr)

# One-vs-one classifier
ovo = OneVsOneClassifier(SVC(kernel='linear', probability=True))
ovo.fit(X_train, y_train)
y_pred_ovo = ovo.predict(X_test)
acc_ovo = accuracy_score(y_test, y_pred_ovo)

print(f"One-vs-rest accuracy: {acc_ovr:.2f}")
print(f"One-vs-one accuracy: {acc_ovo:.2f}")

Important Notes

One-vs-rest is faster when you have many classes because it trains fewer models.

One-vs-one can be more accurate but slower because it trains many models.

Both strategies turn a many-class problem into multiple two-class problems.

Summary

One-vs-rest trains one model per class against all others.

One-vs-one trains one model for every pair of classes.

These strategies help use simple two-class models for many-class problems.