Factual consistency checking in Prompt Engineering / GenAI - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

What is the main goal of factual consistency checking in AI-generated text?

Choose the best description of what factual consistency checking aims to achieve in AI-generated content.

A Maximizing the length of the generated text regardless of content.

B Ensuring the generated text matches the facts in the source or knowledge base.

C Improving the creativity and style of the generated text without regard to facts.

D Reducing the computational cost of generating text.

💡 Hint

Think about why we want AI to be truthful and accurate.

Factual consistency checking ensures that AI-generated text does not contradict known facts or the source material. This helps maintain trust and reliability.

❓ Predict Output

intermediate

Output of a simple factual consistency check using cosine similarity

Given the following Python code snippet that compares embeddings of a source and generated sentence, what is the printed output?

Prompt Engineering / GenAI

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

source_embedding = np.array([[0.1, 0.3, 0.5]])
generated_embedding = np.array([[0.1, 0.3, 0.5]])
similarity = cosine_similarity(source_embedding, generated_embedding)[0][0]
print(round(similarity, 2))

A 1.0

B 0.0

C 0.5

D Error: cosine_similarity requires 2D arrays

💡 Hint

Cosine similarity of identical vectors is 1.

The cosine similarity between two identical vectors is 1, indicating perfect similarity.

❓ Model Choice

advanced

Best model type for factual consistency checking in text generation

Which model type is most suitable for checking factual consistency between a source document and generated summary?

A Binary classifier trained to predict if generated text is factually consistent with source

B Sequence-to-sequence model trained for summarization

C Unsupervised clustering model grouping similar documents

D Generative adversarial network (GAN) for image generation

💡 Hint

Think about a model that can say yes or no about factual correctness.

A binary classifier trained on pairs of source and generated text to predict factual consistency is best suited for this task.

❓ Metrics

advanced

Which metric directly measures factual consistency in generated text?

Among these metrics, which one is designed specifically to evaluate factual consistency between generated text and source content?

A BLEU score

B Perplexity

C FactCC score

D ROUGE score

💡 Hint

This metric was created for factual consistency evaluation.

FactCC is a metric and model designed specifically to detect factual inconsistencies in generated summaries.

🔧 Debug

expert

Why does this factual consistency check code fail to detect errors?

Consider this Python code snippet that compares source and generated sentences using exact string match for factual consistency. Why might it fail to detect factual errors?

Prompt Engineering / GenAI

source = "The Eiffel Tower is in Paris."
generated = "The Eiffel Tower is in Berlin."
consistent = (source == generated)
print(consistent)

A It always returns True because variables are assigned incorrectly.

B It raises a TypeError because strings cannot be compared with '=='.

C It uses cosine similarity which is not suitable for strings.

D It only checks exact string equality, so it misses factual differences in similar sentences.

💡 Hint

Think about what comparing two sentences by '==' really means.

Checking exact string equality does not capture factual differences if sentences differ slightly in wording but have different facts.