Factual consistency checking in Prompt Engineering / GenAI - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to load a pretrained model for factual consistency checking.

Prompt Engineering / GenAI

from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained([1])

Drag options to blanks, or click blank then click option'

A"facebook/bart-large-mnli"

B"roberta-base"

C"gpt2"

D"bert-base-uncased"

Attempts:

3 left

💡 Hint

Common Mistakes

Choosing a language model not fine-tuned for classification.

Using a base model without a classification head.

The 'facebook/bart-large-mnli' model is commonly used for factual consistency checking tasks.

2fill in blank

medium

Complete the code to tokenize input text for factual consistency checking.

Prompt Engineering / GenAI

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-mnli")
inputs = tokenizer([1], return_tensors="pt", truncation=True)

Drag options to blanks, or click blank then click option'

A"This is a claim."

B"This is a document."

C"Hello world!"

D["This is a claim.", "This is a document."]

Attempts:

3 left

💡 Hint

Common Mistakes

Tokenizing only the claim or only the document.

Passing a single string instead of a list.

For factual consistency, we tokenize both the claim and the document as a list of strings.

3fill in blank

hard

Fix the error in the code to get model predictions for factual consistency.

Prompt Engineering / GenAI

outputs = model(**[1])
predictions = outputs.logits.argmax(dim=1)

Drag options to blanks, or click blank then click option'

Atokenizer

Binputs

Coutputs

Dmodel

Attempts:

3 left

💡 Hint

Common Mistakes

Passing the tokenizer object instead of tokenized inputs.

Passing the model or outputs variable.

The model expects tokenized inputs, so we pass 'inputs' as keyword arguments.

4fill in blank

hard

Fill both blanks to create a function that checks if a claim is factually consistent with a document.

Prompt Engineering / GenAI

def check_consistency(claim, document):
    inputs = tokenizer([claim, document], return_tensors=[1], truncation=True)
    outputs = model(**inputs)
    pred = outputs.logits.argmax(dim=[2]).item()
    return pred == 2

Drag options to blanks, or click blank then click option'

A"pt"

B"tf"

C0

D1

Attempts:

3 left

💡 Hint

Common Mistakes

Using TensorFlow tensors when model is PyTorch.

Argmax over wrong dimension.

We use 'pt' for PyTorch tensors and argmax over dimension 1 for batch dimension.

5fill in blank

hard

Fill all three blanks to compute accuracy of factual consistency predictions.

Prompt Engineering / GenAI

correct = 0
for claim, doc, label in data:
    inputs = tokenizer([claim, doc], return_tensors=[1], truncation=True)
    outputs = model(**inputs)
    pred = outputs.logits.argmax(dim=[2]).item()
    if pred == label:
        correct += [3]
accuracy = correct / len(data)

Drag options to blanks, or click blank then click option'

A"pt"

B1

C0

Attempts:

3 left

💡 Hint

Common Mistakes

Using wrong tensor type.

Argmax over wrong dimension.

Incrementing correct by 0 or wrong value.

Use PyTorch tensors, argmax over dimension 1, and increment correct by 1 for each correct prediction.

Practice

1. What is the main purpose of factual consistency checking in AI-generated text?

easy

A. To reduce the size of the AI model

B. To improve the speed of AI text generation

C. To make AI text more creative and imaginative

D. To ensure the AI's output matches true and reliable information

Solution

1. Step 1: Understand the goal of factual consistency checking

   It is used to verify that AI-generated text is accurate and trustworthy.

2. Step 2: Compare options with this goal

   Only To ensure the AI's output matches true and reliable information talks about matching output with true information, which fits the goal.

3. Final Answer:

   To ensure the AI's output matches true and reliable information -> Option D

4. Quick Check:

   Purpose = Verify truthfulness [OK]

Hint: Check which option talks about truth and reliability [OK]

Common Mistakes:

• Confusing creativity with factual accuracy
• Thinking speed or size relates to factual checking
• Ignoring the need for truth in AI outputs

2. Which of the following is a correct simple method for factual consistency checking?

easy

A. Using word overlap between generated text and reference text

B. Training a new AI model from scratch

C. Increasing the number of layers in the AI model

D. Reducing the vocabulary size of the AI

Solution

1. Step 1: Identify simple factual checking methods

   Simple methods often compare words between generated and trusted texts.

2. Step 2: Match options to this method

   Using word overlap between generated text and reference text describes word overlap, a known simple method. Others relate to model design, not checking.

3. Final Answer:

   Using word overlap between generated text and reference text -> Option A

4. Quick Check:

   Simple method = Word overlap [OK]

Hint: Look for word comparison methods, not model changes [OK]

Common Mistakes:

• Confusing model training with checking methods
• Choosing options about model size or layers
• Ignoring the comparison aspect of checking

3. Given the generated sentence: 'The Eiffel Tower is in Berlin.' and the reference sentence: 'The Eiffel Tower is in Paris.', which factual consistency check result is correct?

medium

A. The sentences are factually consistent because they share many words.

B. The sentences are inconsistent because they have different lengths.

C. The sentences are factually inconsistent because the location is different.

D. The sentences are consistent because both mention the Eiffel Tower.

Solution

1. Step 1: Compare key facts in both sentences

   Both mention Eiffel Tower, but locations differ: Berlin vs Paris.

2. Step 2: Determine factual consistency

   Different locations mean factual inconsistency despite word overlap.

3. Final Answer:

   The sentences are factually inconsistent because the location is different. -> Option C

4. Quick Check:

   Location mismatch = Inconsistent [OK]

Hint: Focus on key fact differences, not just shared words [OK]

Common Mistakes:

• Assuming word overlap means consistency
• Ignoring critical fact differences
• Confusing sentence length with factual accuracy

4. You have a simple factual consistency checker that counts overlapping words. It incorrectly marks 'The capital of France is Paris.' and 'Paris is the capital of France.' as inconsistent. What is the likely error?

medium

A. The checker does not ignore word order, causing false inconsistency

B. The checker uses AI understanding, which is too strict

C. The checker compares sentence lengths only

D. The checker ignores common words like 'the' and 'is'

Solution

1. Step 1: Analyze the checker behavior

   It counts overlapping words but marks reordered sentences inconsistent.

2. Step 2: Identify the cause

   Not ignoring word order causes false negatives despite same words.

3. Final Answer:

   The checker does not ignore word order, causing false inconsistency -> Option A

4. Quick Check:

   Word order sensitivity = False inconsistency [OK]

Hint: Check if word order affects overlap counting [OK]

Common Mistakes:

• Assuming AI understanding causes error here
• Thinking sentence length matters
• Ignoring the role of stop words

5. You want to improve factual consistency checking by combining word overlap with AI understanding. Which approach best achieves this?

hard

A. Only count exact word matches without context

B. Use a model that compares semantic meaning, then verify key facts match

C. Ignore reference text and trust AI output blindly

D. Reduce the AI model size to speed up checking

Solution

1. Step 1: Understand combining methods

   Combining word overlap with AI understanding means checking meaning and facts.

2. Step 2: Evaluate options

   Use a model that compares semantic meaning, then verify key facts match uses semantic comparison and fact verification, best for improved checking.

3. Final Answer:

   Use a model that compares semantic meaning, then verify key facts match -> Option B

4. Quick Check:

   Semantic + fact check = Best approach [OK]

Hint: Pick option combining meaning and fact verification [OK]

Common Mistakes:

• Choosing only word matching without context
• Ignoring reference text
• Focusing on model size instead of accuracy