Automated evaluation metrics in Prompt Engineering / GenAI - Cheat Sheet & Quick Revision

Recall & Review

beginner

What are automated evaluation metrics in machine learning?

Automated evaluation metrics are tools that measure how well a machine learning model performs without human judgment. They give quick, objective scores like accuracy or error rates.

beginner

Explain the difference between accuracy and precision.

Accuracy measures how many predictions are correct overall. Precision measures how many predicted positives are actually correct. Accuracy looks at all predictions; precision focuses on positive predictions.

intermediate

What is the F1 score and why is it useful?

The F1 score combines precision and recall into one number. It is useful when you want a balance between catching positives and avoiding false alarms, especially with uneven class sizes.

intermediate

How does Mean Squared Error (MSE) evaluate a regression model?

MSE calculates the average of the squares of the differences between predicted and actual values. It shows how far predictions are from true values, with bigger errors penalized more.

beginner

Why are automated evaluation metrics important in AI development?

They provide fast, consistent, and objective ways to check if models work well. This helps developers improve models and compare different approaches without bias.

Which metric measures the proportion of correct positive predictions out of all positive predictions?

A Precision

B Accuracy

C Recall

D Mean Squared Error

Precision tells us how many predicted positives are actually correct, focusing on the quality of positive predictions.

What does a high Mean Squared Error (MSE) indicate in a regression model?

A Model has high precision

B Predictions are very close to actual values

C Predictions are far from actual values

D Model has balanced recall and precision

High MSE means the average squared difference between predicted and actual values is large, so predictions are far off.

Which metric is best when you want to balance catching positives and avoiding false alarms?

A Accuracy

B Recall

C Mean Absolute Error

D F1 Score

F1 Score combines precision and recall to balance between detecting positives and limiting false positives.

Accuracy is defined as:

A Correct predictions divided by total predictions

B Correct positive predictions divided by all positive predictions

C Correct positive predictions divided by all actual positives

D Average squared difference between predicted and actual values

Accuracy measures the overall correctness of predictions out of all predictions made.

Why do developers use automated evaluation metrics?

A To manually check each prediction

B To get fast and objective model performance scores

C To replace the need for data

D To make models slower

Automated metrics provide quick, unbiased scores to help improve and compare models efficiently.

Describe three common automated evaluation metrics and what they measure.

Think about metrics for classification models.

Explain why automated evaluation metrics are useful when training machine learning models.

Consider how metrics help developers during model building.