Concept drift detection in MLOps - Step-by-Step Execution

Process Flow - Concept drift detection

Start Monitoring Data Stream

↓

Collect New Data Batch

↓

Compare New Data to Training Data

↓

Calculate Drift Metric

↓

Is Drift Detected?

No→Continue Monitoring

Yes↓

Trigger Alert or Retrain Model

↓

Update Model or Data Pipeline

↓

Resume Monitoring

The flow shows how new data is monitored continuously, compared to original training data, and if drift is detected, an alert or retraining is triggered before resuming monitoring.

Execution Sample

MLOps

def detect_drift(old_data, new_data, threshold=0.1):
    drift_score = calculate_statistical_distance(old_data, new_data)
    if drift_score > threshold:
        return True
    return False

This function compares old and new data using a statistical distance and returns True if drift is detected above a threshold.

Process Table

Step	Action	Old Data Summary	New Data Summary	Drift Score	Drift Detected	Next Step
1	Collect new data batch	Mean=50, Std=5	Mean=52, Std=5	0.04	No	Continue Monitoring
2	Collect new data batch	Mean=50, Std=5	Mean=55, Std=6	0.12	Yes	Trigger Alert
3	Retrain model	Model updated with new data	-	-	-	Resume Monitoring
4	Collect new data batch	Mean=55, Std=6	Mean=54, Std=6	0.02	No	Continue Monitoring

💡 Monitoring continues until drift score exceeds threshold, then model update occurs and monitoring resumes.

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4
old_data_summary	Mean=50, Std=5	Mean=50, Std=5	Mean=50, Std=5	Mean=55, Std=6	Mean=55, Std=6
new_data_summary	-	Mean=52, Std=5	Mean=55, Std=6	-	Mean=54, Std=6
drift_score	-	0.04	0.12	-	0.02
drift_detected	-	No	Yes	-	No

Key Moments - 3 Insights

Why does the drift score sometimes increase even if the mean changes only slightly?

Because drift score measures statistical distance considering both mean and standard deviation, small changes in either can increase the score as shown in Step 2 of the execution_table.

What happens after drift is detected at Step 2?

The system triggers an alert and retrains the model with new data, updating old_data_summary as shown in Step 3 before resuming monitoring.

Why does monitoring continue after retraining the model?

Because concept drift can happen anytime, continuous monitoring ensures the model stays accurate, as shown by Step 4 where monitoring resumes with updated data.

Visual Quiz - 3 Questions

Look at the execution_table at Step 2, what is the drift score and was drift detected?

A Drift score is 0.02 and drift detected is No

B Drift score is 0.12 and drift detected is Yes

C Drift score is 0.04 and drift detected is No

D Drift score is 0.12 and drift detected is No

💡 Hint

Check the 'Drift Score' and 'Drift Detected' columns at Step 2 in the execution_table.

At which step does the model get retrained according to the execution_table?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Look for the action 'Retrain model' in the 'Action' column.

If the threshold was lowered to 0.03, at which step would drift be detected first?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Compare drift scores in the execution_table to the new threshold of 0.03.

Concept Snapshot

Concept Drift Detection:
- Continuously monitor new data against training data
- Calculate a drift metric (e.g., statistical distance)
- If drift metric > threshold, trigger alert or retrain
- Update model or pipeline accordingly
- Resume monitoring to catch future drifts

Full Transcript

Concept drift detection means watching new data over time to see if it changes from the data the model learned on. We collect new data batches and compare their statistics to the old data. We calculate a drift score that measures how different the new data is. If this score is above a set threshold, we say drift is detected. Then, we alert or retrain the model with new data to keep it accurate. After updating, we keep watching for more drift. This process helps models stay reliable as data changes.