Perplexity for research and fact-checking in AI for Everyone - Mini Project: Build & Apply

Understanding Perplexity for Research and Fact-Checking

📖 Scenario: You are a researcher who wants to understand how to measure the quality of language models when they help with research and fact-checking.

🎯 Goal: Build a simple explanation and example of how perplexity works and why it matters for checking facts and research accuracy.

📋 What You'll Learn

Create a dictionary called sentence_probabilities with exact word probabilities

Add a variable called total_words with the exact number of words

Calculate the perplexity using the formula with a for loop over sentence_probabilities

Add a final variable called perplexity that holds the calculated value

💡 Why This Matters

🌍 Real World

Perplexity helps researchers understand how well a language model predicts text, which is important for fact-checking and generating accurate information.

💼 Career

Data scientists and AI researchers use perplexity to evaluate and improve language models used in search engines, chatbots, and automated fact-checking tools.

1

Create word probabilities dictionary

Create a dictionary called sentence_probabilities with these exact entries: 'the': 0.1, 'cat': 0.05, 'sat': 0.02, 'on': 0.03, 'mat': 0.01.

AI for Everyone

# Create the dictionary sentence_probabilities with given word probabilities
# Your code here

Hint

Use curly braces to create a dictionary with the exact word-probability pairs.

2

Add total words variable

Add a variable called total_words and set it to the exact number 5, representing the total words in the sentence.

AI for Everyone

sentence_probabilities = {'the': 0.1, 'cat': 0.05, 'sat': 0.02, 'on': 0.03, 'mat': 0.01}
# Add the total_words variable below
# Your code here

Hint

Count the words in the dictionary and assign that number to total_words.

3

Calculate perplexity using loop

Create a variable called product_prob and set it to 1. Then use a for loop with variables word and prob to iterate over sentence_probabilities.items(). Inside the loop, multiply product_prob by prob.

AI for Everyone

sentence_probabilities = {'the': 0.1, 'cat': 0.05, 'sat': 0.02, 'on': 0.03, 'mat': 0.01}
total_words = 5
# Calculate product_prob by multiplying all probabilities
# Your code here

Hint

Start with 1 and multiply by each probability in the dictionary using a loop.

4

Compute final perplexity value

Add a variable called perplexity and set it to product_prob raised to the power of -1 / total_words.

AI for Everyone

sentence_probabilities = {'the': 0.1, 'cat': 0.05, 'sat': 0.02, 'on': 0.03, 'mat': 0.01}
total_words = 5
product_prob = 1
for word, prob in sentence_probabilities.items():
    product_prob *= prob
# Calculate perplexity below
# Your code here

Hint

Use the formula: perplexity = product_prob to the power of (-1 divided by total_words).

Practice

1. What does a low perplexity score indicate about an AI's understanding of text?

easy

A. The AI is confused and predicts text poorly

B. The AI generates random text without meaning

C. The AI ignores the text completely

D. The AI predicts the text well and understands it better

Solution

1. Step 1: Understand what perplexity measures

   Perplexity measures how surprised an AI is by the text it predicts; lower means less surprise.

2. Step 2: Interpret low perplexity meaning

   Low perplexity means the AI predicts the text well, showing better understanding.

3. Final Answer:

   The AI predicts the text well and understands it better -> Option D

4. Quick Check:

   Low perplexity = better prediction [OK]

Hint: Low perplexity means better prediction accuracy [OK]

Common Mistakes:

• Confusing low perplexity with confusion
• Thinking low perplexity means ignoring text
• Assuming low perplexity means random output

2. Which of the following best describes how perplexity is calculated?

easy

A. By measuring the probability of each word predicted by the AI

B. By counting the number of words in a text

C. By checking the length of the AI's output

D. By counting the number of sentences in the text

Solution

1. Step 1: Recall perplexity calculation basics

   Perplexity uses the probabilities the AI assigns to each predicted word to measure surprise.

2. Step 2: Identify correct calculation method

   It is not about counting words or sentences but about the likelihood of predicted words.

3. Final Answer:

   By measuring the probability of each word predicted by the AI -> Option A

4. Quick Check:

   Perplexity = word prediction probabilities [OK]

Hint: Perplexity uses word probabilities, not counts [OK]

Common Mistakes:

• Thinking perplexity counts words or sentences
• Confusing output length with perplexity
• Ignoring probability in calculation

3. Given an AI model with perplexity scores on two texts: Text A = 15, Text B = 50. Which text does the AI understand better?

medium

A. Text B, because higher perplexity means better understanding

B. Text A, because lower perplexity means better understanding

C. Both texts are understood equally

D. Cannot tell from perplexity scores

Solution

1. Step 1: Compare perplexity scores

   Lower perplexity indicates better prediction and understanding by the AI.

2. Step 2: Identify which text has lower perplexity

   Text A has perplexity 15, which is lower than Text B's 50.

3. Final Answer:

   Text A, because lower perplexity means better understanding -> Option B

4. Quick Check:

   Lower perplexity = better understanding [OK]

Hint: Lower perplexity means better AI understanding [OK]

Common Mistakes:

• Assuming higher perplexity means better understanding
• Thinking perplexity scores are unrelated to understanding
• Ignoring the numeric difference in scores

4. An AI researcher notices the perplexity score is unexpectedly high on a simple text. What could be a likely cause?

medium

A. The AI model is not trained well on that type of text

B. The text is too short to calculate perplexity

C. The AI model always produces low perplexity scores

D. Perplexity scores do not depend on the AI model

Solution

1. Step 1: Understand what high perplexity means

   High perplexity means the AI is surprised and predicts poorly.

2. Step 2: Identify cause for high perplexity on simple text

   If the text is simple but perplexity is high, likely the AI model lacks proper training on that text type.

3. Final Answer:

   The AI model is not trained well on that type of text -> Option A

4. Quick Check:

   High perplexity = poor training [OK]

Hint: High perplexity often means poor model training [OK]

Common Mistakes:

• Thinking text length alone causes high perplexity
• Assuming AI always has low perplexity
• Believing perplexity is unrelated to model quality

5. How can perplexity help in fact-checking research when using AI-generated text?

hard

A. By automatically correcting all errors in the text

B. By counting the number of facts in the text

C. By showing how confidently AI predicts text, helping identify reliable information

D. By ignoring the text and focusing on images only

Solution

1. Step 1: Understand perplexity's role in AI text prediction

   Perplexity measures AI confidence in predicting text, indicating reliability.

2. Step 2: Connect perplexity to fact-checking

   Lower perplexity suggests AI is more confident and likely accurate, aiding fact-checking.

3. Final Answer:

   By showing how confidently AI predicts text, helping identify reliable information -> Option C

4. Quick Check:

   Perplexity indicates AI confidence for fact-checking [OK]

Hint: Use low perplexity to spot reliable AI text [OK]

Common Mistakes:

• Thinking perplexity counts facts directly
• Assuming perplexity fixes errors automatically
• Ignoring text and focusing on unrelated data