Experiment - Monitoring model performance

Problem:You have trained a classification model, but you do not have a system to track how well it performs over time on new data.

Current Metrics:Training accuracy: 92%, Validation accuracy: 88%, No monitoring in place.

Issue:Without monitoring, model performance may degrade unnoticed due to data changes or other factors.

Your Task

Implement a simple monitoring system that tracks model accuracy on new incoming data batches and alerts when accuracy drops below 85%.

Use only Python and scikit-learn.

Simulate new data batches for monitoring.

Do not retrain the model during monitoring.

Hint 1

Hint 2

Hint 3

Solution

ML Python

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
import numpy as np

# Step 1: Train initial model
X, y = make_classification(n_samples=1000, n_features=20, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)
model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)

# Step 2: Check initial validation accuracy
val_preds = model.predict(X_val)
val_accuracy = accuracy_score(y_val, val_preds)
print(f"Initial validation accuracy: {val_accuracy:.2f}")

# Step 3: Define monitoring function

def monitor_model(model, batches, threshold=0.85):
    accuracies = []
    for i, (X_batch, y_batch) in enumerate(batches, 1):
        preds = model.predict(X_batch)
        acc = accuracy_score(y_batch, preds)
        accuracies.append(acc)
        print(f"Batch {i} accuracy: {acc:.2f}")
        if acc < threshold:
            print(f"ALERT: Accuracy below threshold ({threshold}) on batch {i}!")
    return accuracies

# Step 4: Simulate new data batches for monitoring
np.random.seed(42)
batches = []
for _ in range(5):
    X_new, y_new = make_classification(n_samples=100, n_features=20, random_state=np.random.randint(1000))
    batches.append((X_new, y_new))

# Step 5: Run monitoring
monitor_accuracies = monitor_model(model, batches)

Added a function to evaluate model accuracy on new data batches.

Simulated new data batches to mimic incoming data.

Implemented alert printout when accuracy falls below 85%.

Results Interpretation

Before Monitoring: No tracking of model performance on new data, risk of unnoticed accuracy drops.

After Monitoring: Accuracy tracked on each batch; alert raised when accuracy dropped below 85% on batch 3.

Monitoring model performance on new data helps detect when the model starts to perform poorly, enabling timely action to maintain quality.

Bonus Experiment

Extend the monitoring system to log accuracy over time to a CSV file and plot the accuracy trend.

💡 Hint

Use Python's csv module or pandas to save data, and matplotlib to create a line plot.