Document loaders in Prompt Engineering / GenAI - ML Experiment: Train & Evaluate

Experiment - Document loaders

Problem:You want to load text documents into your AI system to prepare data for training or analysis.

Current Metrics:Loading speed: 5 documents per second; Data completeness: 90%; Error rate: 10%

Issue:The document loader misses some content and is slow, causing incomplete data and delays.

Your Task

Improve the document loader to achieve at least 98% data completeness and reduce error rate below 2%, while maintaining or improving loading speed.

You cannot change the document source format.

You must keep the loader compatible with plain text and PDF files.

Hint 1

Hint 2

Hint 3

Solution

Prompt Engineering / GenAI

import os
from typing import List
from pdfminer.high_level import extract_text

def load_text_file(file_path: str) -> str:
    with open(file_path, 'r', encoding='utf-8') as f:
        return f.read()

def load_pdf_file(file_path: str) -> str:
    try:
        text = extract_text(file_path)
        return text
    except Exception as e:
        print(f"Error loading PDF {file_path}: {e}")
        return ""

def load_documents(file_paths: List[str]) -> List[str]:
    documents = []
    for path in file_paths:
        ext = os.path.splitext(path)[1].lower()
        if ext == '.txt':
            try:
                text = load_text_file(path)
                documents.append(text)
            except Exception as e:
                print(f"Error loading text file {path}: {e}")
                documents.append("")
        elif ext == '.pdf':
            text = load_pdf_file(path)
            documents.append(text)
        else:
            print(f"Unsupported file type: {path}")
            documents.append("")
    return documents

# Example usage:
# files = ['doc1.txt', 'doc2.pdf']
# loaded_docs = load_documents(files)
# print(f"Loaded {len(loaded_docs)} documents.")

Added pdfminer.six library to extract text from PDFs accurately.

Implemented error handling for both text and PDF loading to reduce failures.

Separated loading logic by file type for clarity and better maintenance.

Results Interpretation

Before: 5 docs/sec, 90% completeness, 10% errors

After: 6 docs/sec, 98% completeness, 1% errors

Using specialized tools and adding error handling improves data quality and speed in document loading.

Bonus Experiment

Try adding support for loading documents from Microsoft Word (.docx) files.

💡 Hint

Use the python-docx library to extract text from .docx files and integrate it into the loader.

Practice

1. What is the main purpose of a document loader in AI applications?

easy

A. To visualize data in charts and graphs

B. To train AI models directly from raw data

C. To read files and convert their content into a format machines can understand

D. To compress files for storage

Solution

1. Step 1: Understand the role of document loaders

   Document loaders are designed to read files and extract their content in a way that machines can process.

2. Step 2: Differentiate from other tasks

   Training models or visualizing data are separate steps after loading the data.

3. Final Answer:

   To read files and convert their content into a format machines can understand -> Option C

4. Quick Check:

   Document loader = read and convert files [OK]

Hint: Remember: loaders prepare data, not train or visualize [OK]

Common Mistakes:

• Confusing loading with training
• Thinking loaders compress files
• Assuming loaders create visualizations

2. Which of the following is the correct way to load a PDF file using a document loader in Python?

easy

A. loader = ImageLoader('file.pdf')

B. loader = PDFLoader('file.pdf')

C. loader = CSVLoader('file.pdf')

D. loader = TextLoader('file.pdf')

Solution

1. Step 1: Identify the correct loader for PDF files

   PDFLoader is designed specifically to read PDF documents.

2. Step 2: Check other loaders' purposes

   TextLoader is for plain text files, CSVLoader for CSV files, and ImageLoader for images, so they are incorrect for PDFs.

3. Final Answer:

   loader = PDFLoader('file.pdf') -> Option B

4. Quick Check:

   PDF file uses PDFLoader [OK]

Hint: Match loader type to file type exactly [OK]

Common Mistakes:

• Using TextLoader for PDFs
• Confusing CSVLoader with PDFLoader
• Trying to load PDFs as images

3. Given the following Python code snippet, what will be the output type of documents after loading a text file?

from langchain.document_loaders import TextLoader
loader = TextLoader('sample.txt')
documents = loader.load()

medium

A. An integer representing file size

B. A single string with all text combined

C. A dictionary with file metadata

D. A list of Document objects containing the text content

Solution

1. Step 1: Understand what TextLoader.load() returns

   The load() method returns a list of Document objects, each holding part or all of the file's text content.

2. Step 2: Eliminate other options

   It does not return a single string, dictionary, or integer.

3. Final Answer:

   A list of Document objects containing the text content -> Option D

4. Quick Check:

   TextLoader.load() returns list of Documents [OK]

Hint: Loaders return lists of Documents, not raw strings [OK]

Common Mistakes:

• Expecting a single string instead of list
• Thinking output is metadata dictionary
• Confusing output with file size

4. Identify the error in this code snippet for loading a PDF file:

from langchain.document_loaders import PDFLoader
loader = PDFLoader('document.txt')
docs = loader.load()

medium

A. The file extension does not match the loader type

B. Missing parentheses in load method

C. Incorrect import statement for PDFLoader

D. The variable name 'docs' is invalid

Solution

1. Step 1: Check file name and loader compatibility

   PDFLoader expects a PDF file, but the file given is 'document.txt', a text file.

2. Step 2: Verify other code parts

   Parentheses are correct, import is correct, and variable name is valid.

3. Final Answer:

   The file extension does not match the loader type -> Option A

4. Quick Check:

   Loader and file type must match [OK]

Hint: Match file extension to loader type to avoid errors [OK]

Common Mistakes:

• Ignoring file extension mismatch
• Thinking variable names cause errors
• Assuming import is wrong without checking

5. You want to load multiple document types (PDF, TXT, CSV) for an AI model training pipeline. Which approach best handles this using document loaders?

hard

A. Use separate loaders for each file type and combine their outputs into one list

B. Use only TextLoader for all files regardless of type

C. Convert all files to images and use ImageLoader

D. Load only PDF files and ignore others

Solution

1. Step 1: Understand file type differences

   Different file types require different loaders to correctly extract content.

2. Step 2: Combine outputs for unified processing

   Using separate loaders and merging their outputs ensures all data is loaded properly for training.

3. Final Answer:

   Use separate loaders for each file type and combine their outputs into one list -> Option A

4. Quick Check:

   Different loaders + combine outputs = best practice [OK]

Hint: Use correct loader per file, then merge results [OK]

Common Mistakes:

• Using one loader for all file types
• Ignoring non-PDF files
• Converting files unnecessarily to images