Why LLM evaluation ensures quality in Prompt Engineering / GenAI - Test Your Understanding

Practice - 5 Tasks

1fill in blank

easy

Complete the code to print the accuracy of the LLM predictions.

Prompt Engineering / GenAI

accuracy = sum(predictions == labels) / len(labels)
print('Accuracy:', [1])

Drag options to blanks, or click blank then click option'

Aaccuracy

Bpredictions

Clabels

Dlen(predictions)

Attempts:

3 left

💡 Hint

Common Mistakes

Printing predictions or labels instead of accuracy.

Using length of predictions instead of accuracy.

The accuracy variable holds the ratio of correct predictions to total labels, so printing accuracy shows the model's quality.

2fill in blank

medium

Complete the code to calculate the F1 score for LLM evaluation.

Prompt Engineering / GenAI

from sklearn.metrics import f1_score
f1 = f1_score(labels, [1])
print('F1 Score:', f1)

Drag options to blanks, or click blank then click option'

Alabels

Bscores

Caccuracy

Dpredictions

Attempts:

3 left

💡 Hint

Common Mistakes

Passing labels twice instead of predictions.

Passing accuracy or scores which are not label arrays.

The f1_score function compares true labels with predicted labels to compute the F1 score, so predictions is the correct input.

3fill in blank

hard

Fix the error in the code to compute the confusion matrix for LLM outputs.

Prompt Engineering / GenAI

from sklearn.metrics import confusion_matrix
cm = confusion_matrix(labels, [1])
print(cm)

Drag options to blanks, or click blank then click option'

Alabels

Bpredictions

Caccuracy

Dscores

Attempts:

3 left

💡 Hint

Common Mistakes

Passing accuracy or labels instead of predictions.

Confusing scores with predicted labels.

The confusion matrix compares true labels with predicted labels, so predictions must be passed as the second argument.

4fill in blank

hard

Fill both blanks to create a dictionary of word counts from LLM output tokens.

Prompt Engineering / GenAI

word_counts = {word: [1] for word in tokens if word [2] stop_words}

Drag options to blanks, or click blank then click option'

Atokens.count(word)

Bin

Cnot in

Dlen(word)

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'in' instead of 'not in' to filter stop words.

Using len(word) instead of count for frequency.

We count how many times each word appears using tokens.count(word). We only include words that are not in the stop words list to focus on meaningful words.

5fill in blank

hard

Fill all three blanks to filter LLM predictions with confidence above threshold and map to labels.

Prompt Engineering / GenAI

filtered = [1]: label for label, score in zip(labels, scores) if score [2] threshold and label [3] valid_labels}

Drag options to blanks, or click blank then click option'

Alabel

B>

Cin

Dscore

Attempts:

3 left

💡 Hint

Common Mistakes

Using label as key instead of score.

Using '<' instead of '>' for filtering.

Checking label not in valid_labels instead of in.

We use score as the key to map labels. We filter scores greater than the threshold using >. We include only labels that are in the valid labels list.

Practice

1. Why is evaluating a Large Language Model (LLM) important?

easy

A. To check if the model gives good and correct answers

B. To make the model run faster

C. To reduce the size of the model

D. To change the model's programming language

Solution

1. Step 1: Understand the purpose of evaluation

   Evaluation is done to see if the model's answers are accurate and useful.

2. Step 2: Compare options with evaluation goals

   Only To check if the model gives good and correct answers matches the goal of checking answer quality, others are unrelated.

3. Final Answer:

   To check if the model gives good and correct answers -> Option A

4. Quick Check:

   Evaluation = Check answer quality [OK]

Hint: Evaluation means checking answer correctness [OK]

Common Mistakes:

• Thinking evaluation speeds up the model
• Confusing evaluation with model size reduction
• Believing evaluation changes programming language

2. Which of the following is a common metric used to evaluate LLMs?

easy

A. Clock speed

B. Screen resolution

C. File size

D. Accuracy

Solution

1. Step 1: Identify evaluation metrics for LLMs

   Metrics like accuracy measure how correct the model's answers are.

2. Step 2: Eliminate unrelated options

   Clock speed, file size, and screen resolution do not measure model quality.

3. Final Answer:

   Accuracy -> Option D

4. Quick Check:

   Evaluation metric = Accuracy [OK]

Hint: Accuracy measures correctness in evaluation [OK]

Common Mistakes:

• Confusing hardware specs with evaluation metrics
• Choosing unrelated technical terms
• Ignoring common ML metrics

3. Given this evaluation result: accuracy = 0.85, what does it mean about the LLM's answers?

medium

A. The model uses 85% of memory

B. The model runs at 85% speed

C. 85% of the model's answers are correct

D. The model is 85% smaller

Solution

1. Step 1: Understand accuracy meaning

   Accuracy of 0.85 means 85% of predictions are correct.

2. Step 2: Match accuracy to options

   Only 85% of the model's answers are correct correctly describes accuracy as correctness percentage.

3. Final Answer:

   85% of the model's answers are correct -> Option C

4. Quick Check:

   Accuracy 0.85 = 85% correct answers [OK]

Hint: Accuracy shows percent correct answers [OK]

Common Mistakes:

• Mixing accuracy with speed or memory
• Thinking accuracy means model size
• Confusing accuracy with hardware usage

4. An LLM evaluation script returns an error when calculating accuracy. Which fix is most likely correct?

predictions = ['yes', 'no', 'yes']
labels = ['yes', 'yes', 'no']
accuracy = sum(predictions == labels) / len(labels)

medium

A. Change predictions to integers

B. Use a loop or list comprehension to compare elements one by one

C. Remove the division by length

D. Use print instead of sum

Solution

1. Step 1: Identify error cause

   Comparing two lists with == returns False, not element-wise comparison.

2. Step 2: Fix comparison method

   Use a loop or list comprehension to compare each element and sum matches.

3. Final Answer:

   Use a loop or list comprehension to compare elements one by one -> Option B

4. Quick Check:

   Element-wise comparison needed for accuracy [OK]

Hint: Compare elements one by one for accuracy [OK]

Common Mistakes:

• Using == on whole lists
• Changing data types unnecessarily
• Removing division breaks accuracy calculation

5. You want to improve an LLM's quality by evaluating it with user feedback and test data. Which approach best ensures trustworthy improvement?

hard

A. Combine test data accuracy with real user feedback scores

B. Only use test data accuracy ignoring user feedback

C. Only use user feedback ignoring test data

D. Skip evaluation and update model randomly

Solution

1. Step 1: Understand evaluation sources

   Test data gives objective accuracy; user feedback adds real-world quality insight.

2. Step 2: Choose combined approach

   Combining both ensures balanced, trustworthy model improvement.

3. Final Answer:

   Combine test data accuracy with real user feedback scores -> Option A

4. Quick Check:

   Balanced evaluation = Combined metrics [OK]

Hint: Use both test data and user feedback [OK]

Common Mistakes:

• Ignoring user feedback
• Ignoring test data accuracy
• Updating model without evaluation