Human evaluation frameworks in Prompt Engineering / GenAI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Human Evaluation Metrics

Which of the following best describes the purpose of human evaluation in AI model assessment?

A To speed up model training by using human-labeled data only

B To measure how well a model performs on automated benchmarks without human input

C To replace all automated metrics with human feedback exclusively

D To assess the quality of AI outputs based on human judgment and preferences

💡 Hint

Think about why humans are involved in evaluating AI outputs.

Human evaluation is used to judge AI outputs based on human preferences, which automated metrics may not fully capture.

❓ Metrics

intermediate

Interpreting Human Evaluation Scores

In a human evaluation where raters score AI-generated text from 1 (poor) to 5 (excellent), what does an average score of 4.2 indicate?

A The AI outputs are generally rated as high quality by humans

B The AI outputs are mostly rated as poor by humans

C The AI outputs have a wide range of scores with no clear trend

D The AI outputs are rated exactly the same by all raters

💡 Hint

Consider what a score closer to 5 means in a 1 to 5 scale.

An average score of 4.2 means most raters found the outputs to be good or excellent.

❓ Predict Output

advanced

Output of Human Evaluation Aggregation Code

What is the output of this Python code that aggregates human ratings?

Prompt Engineering / GenAI

ratings = {'rater1': [4, 5, 3], 'rater2': [5, 4, 4], 'rater3': [3, 4, 5]}
average_scores = [sum(scores)/len(scores) for scores in zip(*ratings.values())]
print(average_scores)

A [4.0, 4.333333333333333, 4.0]

B [4.0, 4.0, 4.0]

C [3.5, 4.5, 4.0]

D [5.0, 4.0, 3.0]

💡 Hint

Calculate the average for each position across raters.

The code averages the first ratings (4,5,3), second ratings (5,4,4), and third ratings (3,4,5) respectively.

❓ Model Choice

advanced

Choosing a Human Evaluation Framework for Dialogue Systems

You want to evaluate a chatbot's responses for naturalness and relevance. Which human evaluation framework is most suitable?

A Cross-validation on training data splits

B Pairwise comparison where raters choose the better response between two options

C Automated BLEU score calculation without human input

D Confusion matrix analysis of chatbot intents

💡 Hint

Think about how humans can compare two chatbot responses directly.

Pairwise comparison lets humans directly judge which response is better, useful for subjective qualities like naturalness.

🔧 Debug

expert

Debugging Human Evaluation Data Collection Code

What error does this code raise when collecting human ratings?

Prompt Engineering / GenAI

def collect_ratings(responses):
    ratings = {}
    for i, response in enumerate(responses):
        rating = int(input(f"Rate response {i+1} (1-5): "))
        if rating < 1 or rating > 5:
            raise ValueError("Rating must be between 1 and 5")
        ratings[i] = rating
    return ratings

ratings = collect_ratings(['Hi', 'Hello', 'Hey'])
print(ratings)

A IndexError due to wrong loop indexing

B TypeError because input() returns a string

C ValueError if user inputs a number outside 1-5

D No error, code runs correctly

💡 Hint

Consider what happens if the user inputs 0 or 6.

The code raises ValueError if the rating is outside the allowed range.