Document loading and parsing in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - Document loading and parsing

Which metric matters for Document loading and parsing and WHY

When loading and parsing documents for AI models, the key metric is Parsing Accuracy. This measures how correctly the document content is extracted and structured. Good parsing ensures the AI model receives clean, accurate data to learn from or analyze. Without accurate parsing, the model may get wrong or incomplete information, leading to poor results.

Confusion matrix or equivalent visualization

For document parsing, a confusion matrix can show how many document elements were correctly or incorrectly identified. For example, if parsing extracts text blocks, tables, and images, the matrix might look like this:

      | Predicted \ Actual | Text | Table | Image |
      |--------------------|------|-------|-------|
      | Text               | 90   | 5     | 0     |
      | Table              | 3    | 85    | 2     |
      | Image              | 0    | 1     | 95    |
    

This shows how many elements were correctly parsed (diagonal) versus misclassified (off-diagonal).

Precision vs Recall tradeoff with concrete examples

Precision means how many parsed elements are actually correct. Recall means how many real elements were found by the parser.

For example, if the parser finds 100 tables but only 80 are real tables, precision is 80%. If there are 100 tables in the document but the parser finds only 70, recall is 70%.

High precision but low recall means the parser is careful but misses many elements. High recall but low precision means it finds many elements but with many mistakes. Balance depends on use case.

What "good" vs "bad" metric values look like for this use case

Good parsing: Precision and recall above 90%. Most document parts are correctly identified and extracted.

Bad parsing: Precision or recall below 70%. Many elements are missed or wrongly extracted, causing errors downstream.

Metrics pitfalls

• Ignoring partial parsing: Counting only fully parsed documents misses partial errors.
• Data leakage: Using test documents seen during parser training inflates metrics.
• Overfitting: Parser tuned too much on one document type may fail on others.
• Accuracy paradox: High overall accuracy can hide poor parsing of rare but important elements.

Self-check question

Your document parser has 98% accuracy but only 12% recall on tables. Is it good for production? Why not?

Answer: No, because it misses most tables (low recall). Even if overall accuracy is high, missing tables can cause big problems if tables are important for your task.

Key Result

Parsing accuracy, precision, and recall are key to ensure correct and complete document extraction.

Practice

1. What is the main purpose of document loading in AI projects?

easy

A. To clean the data by removing errors

B. To train the AI model with labeled data

C. To visualize the results of the AI model

D. To read text files so the computer can access their content

Solution

1. Step 1: Understand document loading

   Document loading means reading text files so the computer can access the content inside.

2. Step 2: Differentiate from other tasks

   Training models, visualization, and cleaning are different steps after loading the document.

3. Final Answer:

   To read text files so the computer can access their content -> Option D

4. Quick Check:

   Document loading = reading files [OK]

Hint: Loading means reading files into the computer [OK]

Common Mistakes:

• Confusing loading with training the model
• Thinking loading cleans the data
• Mixing loading with visualization

2. Which Python code snippet correctly loads a text file named data.txt into a string variable?

easy

A. with open('data.txt', 'x') as file:
text = file.read()

B. file = open('data.txt', 'w')
text = file.read()

C. with open('data.txt', 'r') as file:
text = file.read()

D. text = open('data.txt').write()

Solution

1. Step 1: Check file mode for reading

   Mode 'r' opens the file for reading, which is needed to load text.

2. Step 2: Use context manager and read method

   Using with open(...) ensures safe file handling, and file.read() reads all content.

3. Final Answer:

   with open('data.txt', 'r') as file: text = file.read() -> Option C

4. Quick Check:

   Open with 'r' and read() = correct loading [OK]

Hint: Use 'r' mode and read() to load text files [OK]

Common Mistakes:

• Using 'w' mode which is for writing, not reading
• Calling write() instead of read()
• Using 'x' mode which is for creating new files

3. What will be the output of this Python code that parses a loaded text?

text = "Hello world! Welcome to AI."
words = text.split()
print(words)

medium

A. ['Hello', 'world', 'Welcome', 'to', 'AI']

B. ['Hello', 'world!', 'Welcome', 'to', 'AI.']

C. ['Hello world! Welcome to AI.']

D. ['H', 'e', 'l', 'l', 'o']

Solution

1. Step 1: Understand split() method

   The split() method splits the string by spaces into a list of words, keeping punctuation attached.

2. Step 2: Apply split() to the text

   Splitting "Hello world! Welcome to AI." results in ['Hello', 'world!', 'Welcome', 'to', 'AI.'] including punctuation.

3. Final Answer:

   ['Hello', 'world!', 'Welcome', 'to', 'AI.'] -> Option B

4. Quick Check:

   split() by space keeps punctuation attached [OK]

Hint: split() breaks text by spaces, punctuation stays [OK]

Common Mistakes:

• Expecting punctuation to be removed automatically
• Thinking split() returns a single string list
• Confusing split() with list(text) which splits characters

4. Identify the error in this code that tries to parse a document into sentences:

text = "AI is fun. Let's learn it."
sentences = text.split('. ')
print(sentences)

medium

A. The split delimiter '. ' misses the last sentence ending

B. The code should use splitlines() instead of split()

C. The print statement is missing parentheses

D. The variable name 'sentences' is invalid

Solution

1. Step 1: Analyze split delimiter usage

   Splitting by '. ' splits sentences but leaves the last sentence without a trailing '. ' unseparated.

2. Step 2: Understand effect on last sentence

   The last sentence "Let's learn it." remains attached with the period, causing inconsistent splitting.

3. Final Answer:

   The split delimiter '. ' misses the last sentence ending -> Option A

4. Quick Check:

   Splitting by '. ' misses last sentence split [OK]

Hint: Splitting by '. ' misses last sentence if no trailing space [OK]

Common Mistakes:

• Thinking splitlines() splits sentences
• Forgetting print() needs parentheses in Python 3
• Assuming variable names cause errors

5. You have a text file with multiple paragraphs separated by blank lines. Which approach best loads and parses it into a list of paragraphs for AI processing?

hard

A. Read the file, split text by double newlines '\n\n', then strip whitespace from each paragraph

B. Read the file line by line and treat each line as a paragraph

C. Use split() to split by single spaces to get paragraphs

D. Load the file and convert all text to uppercase without splitting

Solution

1. Step 1: Understand paragraph separation

   Paragraphs are separated by blank lines, which means two newline characters '\n\n'.

2. Step 2: Parse paragraphs correctly

   Splitting by '\n\n' divides text into paragraphs; stripping whitespace cleans each paragraph.

3. Final Answer:

   Read the file, split text by double newlines '\n\n', then strip whitespace from each paragraph -> Option A

4. Quick Check:

   Split by '\n\n' for paragraphs [OK]

Hint: Paragraphs split by double newlines '\n\n' [OK]

Common Mistakes:

• Splitting by single spaces splits words, not paragraphs
• Treating each line as a paragraph loses multi-line paragraphs
• Ignoring whitespace cleanup after splitting