Automated evaluation metrics in Prompt Engineering / GenAI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding BLEU score purpose

What is the main purpose of the BLEU score in evaluating machine learning models?

A To measure the similarity between generated text and reference text by comparing overlapping n-grams

B To calculate the accuracy of classification models by counting correct predictions

C To evaluate the speed of model training in seconds

D To measure the memory usage of a model during inference

💡 Hint

Think about how text generation quality is measured by comparing outputs.

BLEU score compares the overlap of small sequences of words (n-grams) between the generated text and reference text to estimate quality.

❓ Predict Output

intermediate

Output of accuracy calculation code

What is the output of the following Python code that calculates accuracy?

Prompt Engineering / GenAI

true_labels = [1, 0, 1, 1, 0]
predictions = [1, 0, 0, 1, 0]
correct = sum(t == p for t, p in zip(true_labels, predictions))
accuracy = correct / len(true_labels)
print(round(accuracy, 2))

A 0.80

B 0.60

C 0.75

D 0.50

💡 Hint

Count how many predictions match the true labels, then divide by total.

There are 4 matches out of 5, so accuracy is 4/5 = 0.8.

❓ Model Choice

advanced

Best metric for imbalanced classification

You have a dataset where one class is very rare. Which metric is best to evaluate your model's performance?

A Accuracy

B BLEU score

C Mean Squared Error

D Precision and Recall

💡 Hint

Think about metrics that focus on positive class detection in imbalanced data.

Precision and recall help measure how well the model detects rare classes, unlike accuracy which can be misleading.

❓ Hyperparameter

advanced

Effect of changing threshold on F1 score

In a binary classifier, what happens to the F1 score if you increase the decision threshold too high?

A F1 score increases because precision and recall both improve

B F1 score becomes undefined

C F1 score decreases because recall drops while precision may increase

D F1 score stays the same regardless of threshold

💡 Hint

Think about how raising threshold affects false negatives and false positives.

Increasing threshold usually reduces recall (more false negatives) and may increase precision, lowering F1 score overall.

🔧 Debug

expert

Identify error in metric calculation code

What error does this code raise when calculating the mean squared error (MSE)?

Prompt Engineering / GenAI

import numpy as np
true = np.array([1, 2, 3])
pred = np.array([1, 2])
mse = np.mean((true - pred) ** 2)
print(mse)

A TypeError because numpy arrays cannot be subtracted

B ValueError due to shape mismatch in subtraction

C ZeroDivisionError when calculating mean

D No error, outputs 0.5

💡 Hint

Check if the arrays have the same length before subtracting.

Subtracting arrays of different shapes causes a ValueError due to shape mismatch.