Evidently AI for monitoring in MLOps - Step-by-Step Execution

Process Flow - Evidently AI for monitoring

Start Data Stream

↓

Collect Data Batch

↓

Feed Data to Evidently AI

↓

Evidently AI Analyzes Data

↓

Generate Monitoring Report

↓

Check for Drift or Anomalies?

No→Continue Monitoring

Yes↓

Trigger Alert or Action

↓

Log and Visualize Results

↓

Repeat for Next Batch

Data flows from collection to Evidently AI analysis, which detects issues and triggers alerts if needed, then logs results for continuous monitoring.

Execution Sample

MLOps

from evidently.dashboard import Dashboard
from evidently.tabs import DataDriftTab

# Create dashboard
dashboard = Dashboard(tabs=[DataDriftTab()])

# Run on data
dashboard.calculate(reference_data, current_data)

# Save report
dashboard.save('report.html')

This code creates a data drift monitoring dashboard with Evidently AI, analyzes reference and current data, then saves an HTML report.

Process Table

Step	Action	Input Data	Evidently AI Process	Output/Result
1	Start monitoring	Reference and current datasets	Initialize dashboard with DataDriftTab	Dashboard object ready
2	Calculate metrics	reference_data, current_data	Compare distributions, detect drift	Drift metrics computed
3	Generate report	Drift metrics	Create visual report with charts	HTML report generated
4	Check drift status	Drift metrics	Evaluate if drift exceeds threshold	Drift detected: Yes/No
5	Trigger alert if needed	Drift detected: Yes	Send alert or log event	Alert sent or logged
6	Save report	HTML report	Write report to file system	report.html saved
7	End of batch	N/A	Wait for next data batch	Monitoring cycle complete

💡 Monitoring cycle ends after report generation and alerting for current data batch

Status Tracker

Variable	Start	After Step 2	After Step 4	Final
dashboard	None	Dashboard object created	Dashboard with metrics	Dashboard saved
drift_metrics	None	Computed metrics	Evaluated for drift	Used for alerting
alert_status	None	None	Yes or No	Alert sent or not

Key Moments - 3 Insights

Why does Evidently AI need both reference and current data?

Evidently AI compares current data against reference data to detect changes or drift, as shown in execution_table step 2 where it calculates metrics by comparing both datasets.

What happens if no drift is detected?

If no drift is detected (execution_table step 4), Evidently AI continues monitoring without triggering alerts, as indicated by the 'No' branch in the concept flow.

How does Evidently AI alert users about detected drift?

When drift is detected (step 5), Evidently AI triggers alerts or logs events to notify users, ensuring timely action, as shown in the execution_table.

Visual Quiz - 3 Questions

Look at the execution_table, what is the output after step 3?

A Drift metrics computed

B HTML report generated

C Alert sent

D Dashboard object created

💡 Hint

Refer to the 'Output/Result' column in execution_table row for step 3.

At which step does Evidently AI decide if drift exceeds threshold?

A Step 2

B Step 5

C Step 4

D Step 6

💡 Hint

Check the 'Action' and 'Evidently AI Process' columns in execution_table for step 4.

If no drift is detected, what happens next according to the concept flow?

A Continue monitoring without alert

B Trigger alert

C Save report and stop

D Delete report

💡 Hint

Look at the decision branch in the concept_flow ASCII diagram after 'Check for Drift or Anomalies?'.

Concept Snapshot

Evidently AI monitors data by comparing current data to reference data.
It calculates drift metrics and generates visual reports.
If drift exceeds thresholds, it triggers alerts.
Reports are saved for review.
This cycle repeats for continuous monitoring.

Full Transcript

Evidently AI is a tool used to monitor machine learning data quality by comparing current data against a reference dataset. The process starts by collecting data batches, which are then fed into Evidently AI. The tool analyzes the data to detect drift or anomalies by calculating metrics that compare distributions. It generates a visual report in HTML format showing these metrics. If the detected drift exceeds a set threshold, Evidently AI triggers alerts or logs events to notify users. The report is saved for review. This monitoring cycle repeats continuously for each new data batch to ensure model reliability and data quality.

Practice

1. What is the main purpose of Evidently AI in ML model monitoring?

easy

A. To clean and preprocess raw data before training

B. To train new machine learning models automatically

C. To deploy ML models to production environments

D. To compare old and new data or predictions to detect changes

Solution

1. Step 1: Understand Evidently AI's role

   Evidently AI is designed to monitor ML models by checking if data or predictions have changed over time.

2. Step 2: Identify the main function

   It compares old and new data or predictions to detect data drift or performance issues.

3. Final Answer:

   To compare old and new data or predictions to detect changes -> Option D

4. Quick Check:

   Monitoring = Comparing data changes [OK]

Hint: Evidently AI checks data changes, not training or deployment [OK]

Common Mistakes:

• Confusing monitoring with training models
• Thinking Evidently deploys models
• Assuming it preprocesses data

2. Which of the following is the correct way to create an Evidently dashboard with tabs for data drift and performance?

easy

A. dashboard = Dashboard(tabs=DataDriftTab(), PerformanceTab())

B. dashboard = Dashboard(tabs=[DataDriftTab(), PerformanceTab()])

C. dashboard = Dashboard(tabs=[PerformanceTab, DataDriftTab])

D. dashboard = Dashboard([DataDriftTab(), PerformanceTab()])

Solution

1. Step 1: Review Evidently dashboard syntax

   The Dashboard class expects a list of tab instances passed as the tabs parameter.

2. Step 2: Check correct instantiation

   Tabs must be instantiated with parentheses and passed as a list to the tabs argument.

3. Final Answer:

   dashboard = Dashboard(tabs=[DataDriftTab(), PerformanceTab()]) -> Option B

4. Quick Check:

   Tabs list with instances = dashboard = Dashboard(tabs=[DataDriftTab(), PerformanceTab()]) [OK]

Hint: Tabs must be instances inside a list assigned to tabs parameter [OK]

Common Mistakes:

• Passing classes instead of instances
• Not using a list for tabs
• Incorrect argument syntax

3. Given the following code snippet, what will be the output type of report.save_html('report.html')?

from evidently.dashboard import Dashboard
from evidently.tabs import DataDriftTab

dashboard = Dashboard(tabs=[DataDriftTab()])
report = dashboard.calculate(reference_data, current_data)
report.save_html('report.html')

medium

A. A new HTML file named 'report.html' is created with the dashboard report

B. An error because save_html() returns a string, not a file

C. The report is printed to the console instead of saved

D. Nothing happens because save_html() is not a valid method

Solution

1. Step 1: Understand save_html() method

   The save_html() method saves the dashboard report as an HTML file to the given path.

2. Step 2: Analyze the code behavior

   Calling report.save_html('report.html') creates a file named 'report.html' containing the report content.

3. Final Answer:

   A new HTML file named 'report.html' is created with the dashboard report -> Option A

4. Quick Check:

   save_html() saves file = A new HTML file named 'report.html' is created with the dashboard report [OK]

Hint: save_html() writes an HTML file, does not print or error [OK]

Common Mistakes:

• Thinking save_html() returns a string
• Expecting console output instead of file
• Assuming save_html() is invalid

4. You wrote this code but get an error: TypeError: 'Dashboard' object is not callable. What is the likely cause?

dashboard = Dashboard(tabs=[DataDriftTab(), PerformanceTab()])
report = dashboard(reference_data, current_data)

medium

A. You forgot to import DataDriftTab

B. Tabs must be strings, not instances

C. You should call dashboard.calculate() instead of dashboard()

D. You need to instantiate PerformanceTab with parameters

Solution

1. Step 1: Identify the error cause

   The error says Dashboard object is not callable, meaning dashboard() is invalid syntax.

2. Step 2: Correct method to generate report

   To get a report, you must call dashboard.calculate(reference_data, current_data), not dashboard().

3. Final Answer:

   You should call dashboard.calculate() instead of dashboard() -> Option C

4. Quick Check:

   Use calculate() method to get report [OK]

Hint: Dashboard object is not callable means missing .calculate() [OK]

Common Mistakes:

• Calling dashboard() directly instead of .calculate()
• Assuming tabs must be strings
• Thinking imports cause this error

5. You want to monitor a model's prediction quality over time using Evidently AI. Which combination of tabs should you include in your dashboard to track data drift and model performance together?

hard

A. DataDriftTab and ClassificationPerformanceTab

B. DataQualityTab and RegressionPerformanceTab

C. DataDriftTab and DataQualityTab

D. RegressionPerformanceTab and ClassificationPerformanceTab

Solution

1. Step 1: Identify tabs for data drift and performance

   DataDriftTab monitors changes in input data distribution. ClassificationPerformanceTab tracks model prediction quality for classification tasks.

2. Step 2: Choose correct combination for monitoring

   To monitor both data drift and model performance for classification, use DataDriftTab and ClassificationPerformanceTab together.

3. Final Answer:

   DataDriftTab and ClassificationPerformanceTab -> Option A

4. Quick Check:

   Data drift + classification performance = DataDriftTab and ClassificationPerformanceTab [OK]

Hint: Match tabs to task: DataDrift + ClassificationPerformance for classification [OK]

Common Mistakes:

• Mixing performance tabs for regression with classification
• Using DataQualityTab instead of DataDriftTab
• Choosing two performance tabs without data drift