Evidently AI for monitoring in MLOps - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is Evidently AI used for in machine learning operations?

Evidently AI is used to monitor machine learning models in production. It tracks data quality, model performance, and detects data drift or model drift to ensure models work well over time.

beginner

What does data drift mean in the context of Evidently AI monitoring?

Data drift means the input data distribution changes over time compared to the data the model was trained on. Evidently AI detects this to alert if the model might perform worse.

intermediate

Name two key metrics Evidently AI tracks to monitor model health.

Evidently AI tracks metrics like accuracy (model performance) and feature distribution statistics (data quality). These help detect if the model or data changes.

intermediate

How does Evidently AI help in detecting model drift?

Evidently AI compares model predictions over time to past predictions or ground truth. If predictions change significantly without reason, it signals model drift.

beginner

What is a common way to visualize monitoring results in Evidently AI?

Evidently AI provides interactive dashboards with charts showing data distributions, metrics over time, and alerts. This helps teams quickly understand model status.

What does Evidently AI primarily monitor in ML systems?

A User interface design

B Server uptime

C Database schema changes

D Model performance and data quality

Evidently AI focuses on monitoring model performance and data quality to ensure ML models work well in production.

Which of the following is a sign of data drift?

A Model code is updated

B Input data distribution changes over time

C Server CPU usage spikes

D User clicks increase

Data drift means the input data distribution changes compared to training data, which can affect model accuracy.

How does Evidently AI detect model drift?

A By comparing recent model predictions to past predictions or true labels

B By checking server logs

C By monitoring network traffic

D By analyzing user feedback

Model drift is detected by comparing how model predictions change over time, which Evidently AI tracks.

What kind of output does Evidently AI provide for monitoring?

A Email newsletters

B Plain text logs only

C Interactive dashboards with charts and alerts

D Source code updates

Evidently AI provides dashboards that visualize monitoring metrics and alerts for easy understanding.

Why is monitoring ML models important?

A To detect when models stop working well due to data or environment changes

B To improve website design

C To reduce server costs

D To increase user registrations

Monitoring helps catch problems like data drift or model drift early, so models stay accurate and reliable.

Explain how Evidently AI helps maintain machine learning model quality in production.

Think about what can go wrong with models after deployment and how Evidently AI watches for those issues.

Describe the types of visualizations you might see in an Evidently AI dashboard and their purpose.

Consider how visual tools help teams quickly understand model health and data changes.

Practice

1. What is the main purpose of Evidently AI in ML model monitoring?

easy

A. To clean and preprocess raw data before training

B. To train new machine learning models automatically

C. To deploy ML models to production environments

D. To compare old and new data or predictions to detect changes

Solution

1. Step 1: Understand Evidently AI's role

   Evidently AI is designed to monitor ML models by checking if data or predictions have changed over time.

2. Step 2: Identify the main function

   It compares old and new data or predictions to detect data drift or performance issues.

3. Final Answer:

   To compare old and new data or predictions to detect changes -> Option D

4. Quick Check:

   Monitoring = Comparing data changes [OK]

Hint: Evidently AI checks data changes, not training or deployment [OK]

Common Mistakes:

• Confusing monitoring with training models
• Thinking Evidently deploys models
• Assuming it preprocesses data

2. Which of the following is the correct way to create an Evidently dashboard with tabs for data drift and performance?

easy

A. dashboard = Dashboard(tabs=DataDriftTab(), PerformanceTab())

B. dashboard = Dashboard(tabs=[DataDriftTab(), PerformanceTab()])

C. dashboard = Dashboard(tabs=[PerformanceTab, DataDriftTab])

D. dashboard = Dashboard([DataDriftTab(), PerformanceTab()])

Solution

1. Step 1: Review Evidently dashboard syntax

   The Dashboard class expects a list of tab instances passed as the tabs parameter.

2. Step 2: Check correct instantiation

   Tabs must be instantiated with parentheses and passed as a list to the tabs argument.

3. Final Answer:

   dashboard = Dashboard(tabs=[DataDriftTab(), PerformanceTab()]) -> Option B

4. Quick Check:

   Tabs list with instances = dashboard = Dashboard(tabs=[DataDriftTab(), PerformanceTab()]) [OK]

Hint: Tabs must be instances inside a list assigned to tabs parameter [OK]

Common Mistakes:

• Passing classes instead of instances
• Not using a list for tabs
• Incorrect argument syntax

3. Given the following code snippet, what will be the output type of report.save_html('report.html')?

from evidently.dashboard import Dashboard
from evidently.tabs import DataDriftTab

dashboard = Dashboard(tabs=[DataDriftTab()])
report = dashboard.calculate(reference_data, current_data)
report.save_html('report.html')

medium

A. A new HTML file named 'report.html' is created with the dashboard report

B. An error because save_html() returns a string, not a file

C. The report is printed to the console instead of saved

D. Nothing happens because save_html() is not a valid method

Solution

1. Step 1: Understand save_html() method

   The save_html() method saves the dashboard report as an HTML file to the given path.

2. Step 2: Analyze the code behavior

   Calling report.save_html('report.html') creates a file named 'report.html' containing the report content.

3. Final Answer:

   A new HTML file named 'report.html' is created with the dashboard report -> Option A

4. Quick Check:

   save_html() saves file = A new HTML file named 'report.html' is created with the dashboard report [OK]

Hint: save_html() writes an HTML file, does not print or error [OK]

Common Mistakes:

• Thinking save_html() returns a string
• Expecting console output instead of file
• Assuming save_html() is invalid

4. You wrote this code but get an error: TypeError: 'Dashboard' object is not callable. What is the likely cause?

dashboard = Dashboard(tabs=[DataDriftTab(), PerformanceTab()])
report = dashboard(reference_data, current_data)

medium

A. You forgot to import DataDriftTab

B. Tabs must be strings, not instances

C. You should call dashboard.calculate() instead of dashboard()

D. You need to instantiate PerformanceTab with parameters

Solution

1. Step 1: Identify the error cause

   The error says Dashboard object is not callable, meaning dashboard() is invalid syntax.

2. Step 2: Correct method to generate report

   To get a report, you must call dashboard.calculate(reference_data, current_data), not dashboard().

3. Final Answer:

   You should call dashboard.calculate() instead of dashboard() -> Option C

4. Quick Check:

   Use calculate() method to get report [OK]

Hint: Dashboard object is not callable means missing .calculate() [OK]

Common Mistakes:

• Calling dashboard() directly instead of .calculate()
• Assuming tabs must be strings
• Thinking imports cause this error

5. You want to monitor a model's prediction quality over time using Evidently AI. Which combination of tabs should you include in your dashboard to track data drift and model performance together?

hard

A. DataDriftTab and ClassificationPerformanceTab

B. DataQualityTab and RegressionPerformanceTab

C. DataDriftTab and DataQualityTab

D. RegressionPerformanceTab and ClassificationPerformanceTab

Solution

1. Step 1: Identify tabs for data drift and performance

   DataDriftTab monitors changes in input data distribution. ClassificationPerformanceTab tracks model prediction quality for classification tasks.

2. Step 2: Choose correct combination for monitoring

   To monitor both data drift and model performance for classification, use DataDriftTab and ClassificationPerformanceTab together.

3. Final Answer:

   DataDriftTab and ClassificationPerformanceTab -> Option A

4. Quick Check:

   Data drift + classification performance = DataDriftTab and ClassificationPerformanceTab [OK]

Hint: Match tabs to task: DataDrift + ClassificationPerformance for classification [OK]

Common Mistakes:

• Mixing performance tabs for regression with classification
• Using DataQualityTab instead of DataDriftTab
• Choosing two performance tabs without data drift