Model drift detection in ML Python - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is model drift in machine learning?

Model drift happens when a machine learning model's performance gets worse over time because the data it sees changes from what it was trained on.

beginner

Name two common types of model drift.

The two common types are:<br>1. Concept drift: When the relationship between input and output changes.<br>2. Data drift: When the input data distribution changes but the relationship stays the same.

beginner

Why is detecting model drift important?

Detecting model drift helps keep models accurate and reliable by alerting us when the model needs retraining or updating.

intermediate

What is a simple method to detect data drift?

A simple method is to compare statistics like mean or distribution of new data with the training data using tests like the Kolmogorov-Smirnov test.

intermediate

How can monitoring model performance metrics help in drift detection?

If metrics like accuracy or error rate suddenly change on new data, it can indicate model drift and that the model may no longer be reliable.

What does concept drift mean?

A Model architecture changes

B Change in the input data distribution only

C Change in the relationship between inputs and outputs

D Model training time increases

Concept drift means the way inputs relate to outputs changes, affecting model predictions.

Which metric change might indicate model drift?

A Sudden drop in accuracy

B Increase in training speed

C More features added

D Longer model code

A sudden drop in accuracy on new data suggests the model is not performing well, indicating drift.

Which test can be used to detect data drift?

A T-test for model weights

B Kolmogorov-Smirnov test

C Gradient descent

D Cross-validation

The Kolmogorov-Smirnov test compares distributions to detect changes in data.

What should you do if model drift is detected?

A Delete the model

B Ignore it

C Change the programming language

D Retrain or update the model

Retraining or updating the model helps it adapt to new data and maintain accuracy.

Data drift affects which part of the model?

A Input data distribution

B Model output labels

C Model hyperparameters

D Training algorithm

Data drift means the input data distribution changes over time.

Explain what model drift is and why it matters in simple terms.

Think about how a model can become less accurate over time.

Describe two ways to detect model drift and what actions to take after detection.

Consider checking model results and comparing new data to old data.