Document loading and parsing in Prompt Engineering / GenAI - Model Pipeline Trace

Model Pipeline - Document loading and parsing

This pipeline shows how a document is loaded, cleaned, and transformed into a format that a machine learning model can understand. It helps computers read and learn from text documents.

Data Flow - 4 Stages

1Document Loading

1 document (text file)→Read raw text from file→1 string (full document text)

"The quick brown fox jumps over the lazy dog."

↓

2Text Cleaning

1 string (full document text)→Remove punctuation, lowercase all letters→1 string (cleaned text)

"the quick brown fox jumps over the lazy dog"

↓

3Tokenization

1 string (cleaned text)→Split text into words (tokens)→1 list of tokens

["the", "quick", "brown", "fox", "jumps", "over", "the", "lazy", "dog"]

↓

4Vectorization

1 list of tokens→Convert tokens to numbers (word indices or embeddings)→1 list of vectors or indices

[12, 45, 78, 34, 56, 23, 12, 89, 67]

Training Trace - Epoch by Epoch

Loss
1.0 |***************
0.8 |************  
0.6 |********     
0.4 |******       
0.2 |****         
0.0 +------------
     1 2 3 4 5 Epochs

Epoch	Loss ↓	Accuracy ↑	Observation
1	0.85	0.6	Model starts learning basic patterns from document vectors.
2	0.65	0.72	Loss decreases and accuracy improves as model understands text better.
3	0.5	0.8	Model shows good learning progress on document data.
4	0.4	0.85	Loss continues to decrease, accuracy rises steadily.
5	0.35	0.88	Model converges well on document parsing task.

Prediction Trace - 5 Layers

Layer 1: Input Document

Layer 2: Text Cleaning

Layer 3: Tokenization

Layer 4: Vectorization

Layer 5: Model Prediction

Model Quiz - 3 Questions

What happens during the 'Text Cleaning' stage?

A Removing punctuation and lowercasing text

B Splitting text into words

C Converting words to numbers

D Loading raw text from file

Key Insight

Document loading and parsing transforms raw text into numbers that a model can understand. Cleaning and tokenization prepare the text, and vectorization turns words into numbers. Training improves the model's ability to predict from these numbers, shown by decreasing loss and increasing accuracy.

Practice

1. What is the main purpose of document loading in AI projects?

easy

A. To clean the data by removing errors

B. To train the AI model with labeled data

C. To visualize the results of the AI model

D. To read text files so the computer can access their content

Solution

1. Step 1: Understand document loading

   Document loading means reading text files so the computer can access the content inside.

2. Step 2: Differentiate from other tasks

   Training models, visualization, and cleaning are different steps after loading the document.

3. Final Answer:

   To read text files so the computer can access their content -> Option D

4. Quick Check:

   Document loading = reading files [OK]

Hint: Loading means reading files into the computer [OK]

Common Mistakes:

• Confusing loading with training the model
• Thinking loading cleans the data
• Mixing loading with visualization

2. Which Python code snippet correctly loads a text file named data.txt into a string variable?

easy

A. with open('data.txt', 'x') as file:
text = file.read()

B. file = open('data.txt', 'w')
text = file.read()

C. with open('data.txt', 'r') as file:
text = file.read()

D. text = open('data.txt').write()

Solution

1. Step 1: Check file mode for reading

   Mode 'r' opens the file for reading, which is needed to load text.

2. Step 2: Use context manager and read method

   Using with open(...) ensures safe file handling, and file.read() reads all content.

3. Final Answer:

   with open('data.txt', 'r') as file: text = file.read() -> Option C

4. Quick Check:

   Open with 'r' and read() = correct loading [OK]

Hint: Use 'r' mode and read() to load text files [OK]

Common Mistakes:

• Using 'w' mode which is for writing, not reading
• Calling write() instead of read()
• Using 'x' mode which is for creating new files

3. What will be the output of this Python code that parses a loaded text?

text = "Hello world! Welcome to AI."
words = text.split()
print(words)

medium

A. ['Hello', 'world', 'Welcome', 'to', 'AI']

B. ['Hello', 'world!', 'Welcome', 'to', 'AI.']

C. ['Hello world! Welcome to AI.']

D. ['H', 'e', 'l', 'l', 'o']

Solution

1. Step 1: Understand split() method

   The split() method splits the string by spaces into a list of words, keeping punctuation attached.

2. Step 2: Apply split() to the text

   Splitting "Hello world! Welcome to AI." results in ['Hello', 'world!', 'Welcome', 'to', 'AI.'] including punctuation.

3. Final Answer:

   ['Hello', 'world!', 'Welcome', 'to', 'AI.'] -> Option B

4. Quick Check:

   split() by space keeps punctuation attached [OK]

Hint: split() breaks text by spaces, punctuation stays [OK]

Common Mistakes:

• Expecting punctuation to be removed automatically
• Thinking split() returns a single string list
• Confusing split() with list(text) which splits characters

4. Identify the error in this code that tries to parse a document into sentences:

text = "AI is fun. Let's learn it."
sentences = text.split('. ')
print(sentences)

medium

A. The split delimiter '. ' misses the last sentence ending

B. The code should use splitlines() instead of split()

C. The print statement is missing parentheses

D. The variable name 'sentences' is invalid

Solution

1. Step 1: Analyze split delimiter usage

   Splitting by '. ' splits sentences but leaves the last sentence without a trailing '. ' unseparated.

2. Step 2: Understand effect on last sentence

   The last sentence "Let's learn it." remains attached with the period, causing inconsistent splitting.

3. Final Answer:

   The split delimiter '. ' misses the last sentence ending -> Option A

4. Quick Check:

   Splitting by '. ' misses last sentence split [OK]

Hint: Splitting by '. ' misses last sentence if no trailing space [OK]

Common Mistakes:

• Thinking splitlines() splits sentences
• Forgetting print() needs parentheses in Python 3
• Assuming variable names cause errors

5. You have a text file with multiple paragraphs separated by blank lines. Which approach best loads and parses it into a list of paragraphs for AI processing?

hard

A. Read the file, split text by double newlines '\n\n', then strip whitespace from each paragraph

B. Read the file line by line and treat each line as a paragraph

C. Use split() to split by single spaces to get paragraphs

D. Load the file and convert all text to uppercase without splitting

Solution

1. Step 1: Understand paragraph separation

   Paragraphs are separated by blank lines, which means two newline characters '\n\n'.

2. Step 2: Parse paragraphs correctly

   Splitting by '\n\n' divides text into paragraphs; stripping whitespace cleans each paragraph.

3. Final Answer:

   Read the file, split text by double newlines '\n\n', then strip whitespace from each paragraph -> Option A

4. Quick Check:

   Split by '\n\n' for paragraphs [OK]

Hint: Paragraphs split by double newlines '\n\n' [OK]

Common Mistakes:

• Splitting by single spaces splits words, not paragraphs
• Treating each line as a paragraph loses multi-line paragraphs
• Ignoring whitespace cleanup after splitting