Human evaluation frameworks in Prompt Engineering / GenAI - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the main purpose of human evaluation frameworks in AI?

Human evaluation frameworks help measure how well AI systems perform by using human judgment to assess qualities like accuracy, relevance, and user satisfaction.

beginner

Name two common criteria used in human evaluation frameworks for AI outputs.

Common criteria include fluency (how natural the output sounds) and relevance (how well the output matches the input or task).

intermediate

Why is inter-rater reliability important in human evaluation?

Inter-rater reliability ensures that different human evaluators give consistent scores, making the evaluation results trustworthy and less biased.

beginner

Describe a simple human evaluation method for text generation models.

A simple method is to ask multiple people to rate generated sentences on a scale (e.g., 1 to 5) for qualities like clarity and correctness, then average the scores.

intermediate

What is a limitation of human evaluation frameworks?

They can be time-consuming, costly, and sometimes subjective, which means results might vary depending on who evaluates and when.

What does inter-rater reliability measure in human evaluation?

A Speed of AI model predictions

B Consistency between different human evaluators

C Number of evaluation criteria used

D Accuracy of automated metrics

Inter-rater reliability checks if different people give similar scores, ensuring consistent human evaluation.

Which of the following is NOT a typical criterion in human evaluation of AI outputs?

A Coherence

B Relevance

C Fluency

D Model training time

Model training time is a technical metric, not a human evaluation criterion.

Why might human evaluation be preferred over automated metrics?

A Humans are faster than machines

B Automated metrics are always inaccurate

C Humans can judge quality aspects that machines cannot easily measure

D Human evaluation is cheaper

Humans can understand nuances like meaning and style that automated metrics may miss.

What is a common scale used in human evaluation ratings?

A 1 to 5

B 0 to 1000

C True or False

D A to Z

A 1 to 5 scale is simple and widely used for rating quality.

Which factor can reduce the reliability of human evaluation?

A Different interpretations by evaluators

B Using multiple evaluators

C Clear evaluation guidelines

D Randomizing evaluation order

If evaluators interpret criteria differently, scores become inconsistent.

Explain what human evaluation frameworks are and why they are important in AI.

Think about how humans check AI outputs for quality.

Describe how inter-rater reliability affects the trustworthiness of human evaluation results.

Consider what happens if evaluators disagree a lot.