Evidently AI for monitoring in MLOps - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Evidently AI's primary purpose

What is the main function of Evidently AI in a machine learning workflow?

A To train machine learning models faster using GPU acceleration

B To monitor data and model performance over time to detect drift and anomalies

C To deploy machine learning models as REST APIs automatically

D To generate synthetic data for training machine learning models

💡 Hint

Think about what happens after a model is deployed and needs ongoing checks.

Evidently AI focuses on monitoring data and model quality metrics to detect changes or problems after deployment.

💻 Command Output

intermediate

Evidently AI report generation output

What output will the following Evidently AI Python code produce?

MLOps

from evidently.dashboard import Dashboard
from evidently.tabs import DataDriftTab
import pandas as pd

reference_data = pd.DataFrame({'feature': [1, 2, 3, 4, 5]})
curr_data = pd.DataFrame({'feature': [1, 2, 2, 4, 5]})

dashboard = Dashboard(tabs=[DataDriftTab()])
dashboard.calculate(reference_data, curr_data)
result = dashboard.json()
print(result)

A A runtime error because DataDriftTab requires target data

B A SyntaxError due to missing import statements

C A JSON string summarizing data drift metrics showing slight drift in 'feature'

D An empty JSON string because no drift is detected

💡 Hint

DataDriftTab compares distributions between reference and current data.

The code calculates data drift between two datasets and outputs a JSON report with metrics indicating if drift occurred.

❓ Configuration

advanced

Configuring Evidently AI for model performance monitoring

Which configuration snippet correctly sets up Evidently AI to monitor model prediction quality with a threshold for accuracy drop?

A

from evidently.metric_preset import ClassificationPreset
from evidently import ColumnMapping

column_mapping = ColumnMapping(prediction='pred', target='target')
metrics = ClassificationPreset()
metrics.add_metric('accuracy', threshold=0.8)

B

from evidently.metric_preset import ClassificationPreset
from evidently import ColumnMapping

column_mapping = ColumnMapping(prediction='pred', target='target')
metrics = ClassificationPreset()
metrics.accuracy_threshold = 0.8

C

from evidently.metric_preset import ClassificationPreset
from evidently import ColumnMapping

column_mapping = ColumnMapping(prediction='pred', target='target')
metrics = ClassificationPreset(accuracy_threshold=0.8)

D

from evidently.metric_preset import ClassificationPreset
from evidently import ColumnMapping

column_mapping = ColumnMapping(prediction='pred', target='target')
metrics = ClassificationPreset()
metrics.add_accuracy_threshold(0.8)

💡 Hint

Look for the method that explicitly sets a threshold for accuracy in the preset.

Option D uses the correct method add_accuracy_threshold to set the accuracy threshold in the ClassificationPreset.

❓ Troubleshoot

advanced

Diagnosing Evidently AI dashboard calculation error

You run Evidently AI's dashboard.calculate(reference_data, current_data) but get a KeyError: 'target'. What is the most likely cause?

A The current_data or reference_data is missing the 'target' column required for metrics

B The Python environment is missing the pandas package

C The dashboard object was not initialized with the DataDriftTab tab

D Evidently AI requires the 'target' column to be named 'label' instead

💡 Hint

Check if your data frames contain all columns needed by the metrics.

Many Evidently metrics require a 'target' column to compare predictions against true values. Missing it causes KeyError.

🔀 Workflow

expert

Optimal Evidently AI monitoring workflow integration

Which sequence correctly describes the best workflow to integrate Evidently AI monitoring into a CI/CD pipeline for ML models?

A 1,2,3,4

B 2,1,3,4

C 1,3,2,4

D 3,1,2,4

💡 Hint

Think about the logical order from data preparation to deployment to monitoring.

The correct workflow starts with collecting baseline data, then deploying the model, then collecting production data, and finally running monitoring reports.