Semi-supervised learning basics in ML Python - ML Experiment: Train & Evaluate

Experiment - Semi-supervised learning basics

Problem:You have a small set of labeled data and a larger set of unlabeled data. You want to build a model that learns from both to improve accuracy.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%

Issue:The model overfits the small labeled data and does not generalize well to validation data.

Your Task

Reduce overfitting by using semi-supervised learning techniques to improve validation accuracy to at least 80% while keeping training accuracy below 90%.

You can only modify the model to use both labeled and unlabeled data.

Do not increase the size of labeled data.

Keep the model architecture simple (e.g., a small neural network).

Hint 1

Hint 2

Hint 3

Hint 4

Solution

ML Python

import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import accuracy_score

# Create synthetic data
X, y = make_classification(n_samples=1000, n_features=20, n_informative=15, n_classes=2, random_state=42)

# Split into labeled, unlabeled, and test sets
X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_labeled, X_unlabeled, y_labeled, _ = train_test_split(X_temp, y_temp, test_size=0.8, random_state=42)

# Initial model trained only on labeled data
model = MLPClassifier(hidden_layer_sizes=(50,), max_iter=200, random_state=42)
model.fit(X_labeled, y_labeled)

# Evaluate initial model
train_acc = accuracy_score(y_labeled, model.predict(X_labeled))
val_acc = accuracy_score(y_test, model.predict(X_test))

# Semi-supervised learning with pseudo-labeling
threshold = 0.9
for iteration in range(5):
    # Predict probabilities on unlabeled data
    probs = model.predict_proba(X_unlabeled)
    max_probs = np.max(probs, axis=1)
    pseudo_labels = model.predict(X_unlabeled)

    # Select confident predictions
    confident_mask = max_probs >= threshold
    if not np.any(confident_mask):
        break

    X_pseudo = X_unlabeled[confident_mask]
    y_pseudo = pseudo_labels[confident_mask]

    # Combine labeled and pseudo-labeled data
    X_combined = np.vstack((X_labeled, X_pseudo))
    y_combined = np.hstack((y_labeled, y_pseudo))

    # Remove pseudo-labeled from unlabeled
    X_unlabeled = X_unlabeled[~confident_mask]

    # Retrain model
    model = MLPClassifier(hidden_layer_sizes=(50,), max_iter=200, random_state=42)
    model.fit(X_combined, y_combined)

    # Update labeled data
    X_labeled, y_labeled = X_combined, y_combined

# Final evaluation
final_train_acc = accuracy_score(y_labeled, model.predict(X_labeled))
final_val_acc = accuracy_score(y_test, model.predict(X_test))

print(f"Initial training accuracy: {train_acc:.2f}")
print(f"Initial validation accuracy: {val_acc:.2f}")
print(f"Final training accuracy: {final_train_acc:.2f}")
print(f"Final validation accuracy: {final_val_acc:.2f}")

Added pseudo-labeling to assign labels to unlabeled data based on model confidence.

Selected only confident predictions above a threshold to add to training data.

Retrained the model iteratively with combined labeled and pseudo-labeled data.

This helped the model learn from more data and reduced overfitting.

Results Interpretation

Before: Training accuracy 95%, Validation accuracy 70% (overfitting)

After: Training accuracy 88%, Validation accuracy 82% (better generalization)

Using unlabeled data with pseudo-labeling helps the model learn more patterns and reduces overfitting, improving validation accuracy.

Bonus Experiment

Try using consistency regularization by adding small noise to unlabeled data and enforcing consistent predictions.

💡 Hint

Add noise to inputs and train the model to predict the same label for original and noisy inputs to improve robustness.

Practice

1. What is the main idea behind semi-supervised learning in machine learning?

easy

A. Using only unlabeled data to train a model

B. Using only labeled data to train a model

C. Using both labeled and unlabeled data to train a model

D. Training multiple models independently

Solution

1. Step 1: Understand the data types in semi-supervised learning

   Semi-supervised learning uses a mix of labeled and unlabeled data to improve model training.

2. Step 2: Compare options with the definition

   Using both labeled and unlabeled data to train a model correctly states the use of both labeled and unlabeled data, unlike other options which mention only one type or unrelated concepts.

3. Final Answer:

   Using both labeled and unlabeled data to train a model -> Option C

4. Quick Check:

   Semi-supervised learning = labeled + unlabeled data [OK]

Hint: Remember: semi-supervised = mix of labeled and unlabeled [OK]

Common Mistakes:

• Confusing semi-supervised with supervised learning
• Thinking it uses only unlabeled data
• Assuming it trains multiple models separately

2. Which of the following is a common method used in semi-supervised learning?

easy

A. Self-training

B. Gradient boosting

C. K-means clustering

D. Decision trees

Solution

1. Step 1: Identify methods specific to semi-supervised learning

   Self-training is a popular semi-supervised method where the model labels unlabeled data iteratively.

2. Step 2: Eliminate unrelated methods

   Gradient boosting and decision trees are supervised learning methods; K-means is unsupervised clustering, not semi-supervised.

3. Final Answer:

   Self-training -> Option A

4. Quick Check:

   Semi-supervised method = Self-training [OK]

Hint: Look for methods that use model to label unlabeled data [OK]

Common Mistakes:

• Confusing supervised methods as semi-supervised
• Choosing clustering as semi-supervised
• Not knowing self-training meaning

3. Consider this Python snippet using label spreading for semi-supervised learning:

from sklearn.semi_supervised import LabelSpreading
import numpy as np

X = np.array([[1], [2], [3], [4], [5]])
y = np.array([0, 1, -1, -1, -1])  # -1 means unlabeled

model = LabelSpreading()
model.fit(X, y)
preds = model.transduction_
print(preds)

What will be the output printed by print(preds)?

medium

A. [0 1 0 0 0]

B. [1 1 1 1 1]

C. [0 1 -1 -1 -1]

D. [0 1 1 1 1]

Solution

1. Step 1: Understand label spreading behavior

   Label spreading propagates labels from labeled points (0 and 1) to unlabeled points (-1) based on similarity.

2. Step 2: Predict labels for unlabeled points

   Since points 2,3,4 are close to labeled point 1, they get label 1. Points 0 and 1 keep their labels 0 and 1.

3. Final Answer:

   [0 1 1 1 1] -> Option D

4. Quick Check:

   Label spreading fills unlabeled with nearest labels [OK]

Hint: Label spreading fills unlabeled with nearest known labels [OK]

Common Mistakes:

• Assuming unlabeled points remain -1
• Thinking labels spread to 0 instead of 1
• Confusing output with input labels

4. The following code attempts to use self-training but has an error:

from sklearn.semi_supervised import SelfTrainingClassifier
from sklearn.svm import SVC

X = [[1], [2], [3], [4]]
y = [0, 1, -1, -1]

base_model = SVC()
model = SelfTrainingClassifier(base_model)
model.fit(X, y)

What is the error in this code?

medium

A. Labels cannot contain -1 for unlabeled data

B. SVC requires probability=True for self-training

C. X must be a numpy array, not a list

D. SelfTrainingClassifier cannot use SVC as base model

Solution

1. Step 1: Check requirements for SelfTrainingClassifier base model

   SelfTrainingClassifier needs base model to provide probability estimates, so SVC must be initialized with probability=True.

2. Step 2: Identify the missing argument

   The code uses default SVC without probability=True, causing an error during fit.

3. Final Answer:

   SVC requires probability=True for self-training -> Option B

4. Quick Check:

   SelfTrainingClassifier needs probabilistic base model [OK]

Hint: Remember: SVC needs probability=True for self-training [OK]

Common Mistakes:

• Thinking -1 labels are invalid
• Believing lists can't be used as input
• Assuming SVC can't be base model

5. You have a dataset with 1000 samples but only 50 are labeled. You want to improve model accuracy using semi-supervised learning. Which approach is best to start with?

hard

A. Use self-training with a base classifier that predicts labels on unlabeled data iteratively

B. Ignore unlabeled data and train only on 50 labeled samples

C. Use unsupervised clustering to label all data without any model

D. Label all 950 samples manually before training

Solution

1. Step 1: Understand the problem with few labeled samples

   With only 50 labeled samples, training a model directly may not generalize well.

2. Step 2: Choose a semi-supervised method to leverage unlabeled data

   Self-training uses the base classifier to label unlabeled data iteratively, improving learning without costly manual labeling.

3. Final Answer:

   Use self-training with a base classifier that predicts labels on unlabeled data iteratively -> Option A

4. Quick Check:

   Semi-supervised learning improves with self-training on unlabeled data [OK]

Hint: Start with self-training to use unlabeled data effectively [OK]

Common Mistakes:

• Ignoring unlabeled data wastes valuable information
• Assuming manual labeling is always feasible
• Confusing clustering with semi-supervised learning