Custom document loaders in LangChain - Step-by-Step Execution

Concept Flow - Custom document loaders

Define Custom Loader Class

↓

Implement load() Method

↓

Instantiate Loader Object

↓

Call load() to Read Documents

↓

Return List of Document Objects

↓

Use Documents in LangChain Pipeline

The flow shows creating a custom loader class with a load method, then using it to read and return documents for LangChain.

Execution Sample

LangChain

from langchain.schema import Document

class MyLoader:
    def __init__(self, path):
        self.path = path
    def load(self):
        with open(self.path, 'r') as f:
            text = f.read()
        return [Document(page_content=text)]

This code defines a custom loader that reads a file and returns its content wrapped in a Document object.

Execution Table

Step	Action	Evaluation	Result
1	Create MyLoader instance with path 'file.txt'	path='file.txt'	MyLoader object created with path='file.txt'
2	Call load() method	Open 'file.txt' for reading	File opened successfully
3	Read file content	Read all text from file	Text content stored in variable
4	Create Document object	Wrap text in Document(page_content=text)	List with one Document returned
5	Return documents list	Return [Document]	Documents ready for LangChain use

💡 load() completes after returning list of Document objects

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	Final
self.path	N/A	'file.txt'	'file.txt'	'file.txt'	'file.txt'
f (file handle)	N/A	open file.txt	open file.txt	closed	closed
text	N/A	N/A	'file content string'	'file content string'	'file content string'
documents	N/A	N/A	N/A	[Document(page_content=text)]	[Document(page_content=text)]

Key Moments - 3 Insights

Why do we return a list with Document objects instead of just the text?

LangChain expects a list of Document objects to handle metadata and text uniformly, as shown in execution_table step 4.

What happens if the file path is wrong or file can't be opened?

An error occurs at step 2 when trying to open the file; this stops load() and must be handled outside or with try-except.

Can load() return multiple documents?

Yes, load() can return a list with many Document objects if the source has multiple parts, but here it returns one as in step 4.

Visual Quiz - 3 Questions

Look at the execution table, what is the value of 'text' after step 3?

A File handle object

B 'file content string'

C List of Document objects

D None

💡 Hint

Check the 'text' variable column in variable_tracker after step 3

At which step does the load() method create the Document object?

A Step 2

B Step 3

C Step 4

D Step 5

💡 Hint

See the 'Action' and 'Result' columns in execution_table for step 4

If the file path was invalid, what would happen during execution?

A Error occurs at step 2 opening file

B load() returns empty list

C Document created with empty content

D Execution continues normally

💡 Hint

Refer to key_moments about file opening errors and execution_table step 2

Concept Snapshot

Custom document loaders in LangChain:
- Define a class with a load() method
- load() reads source data (e.g., file)
- Wrap text in Document objects
- Return a list of Documents
- Used to feed data into LangChain pipelines

Full Transcript

This visual execution trace shows how to create and use a custom document loader in LangChain. First, you define a class with an __init__ method to store the file path. Then, the load() method opens the file, reads its content, and wraps it in a Document object. The method returns a list containing this Document. The execution table traces each step from creating the loader instance, opening the file, reading text, creating the Document, and returning the list. The variable tracker shows how variables like path, file handle, text, and documents change during execution. Key moments clarify why we return Document objects, what happens if the file can't be opened, and that load() can return multiple documents. The quiz tests understanding of variable values at steps, when Document is created, and error handling. The snapshot summarizes the pattern for custom loaders in LangChain.

Practice

1. What is the main purpose of creating a custom document loader in Langchain?

easy

A. To load and preprocess documents from any data source before use

B. To create user interfaces for document display

C. To train machine learning models directly

D. To manage database connections automatically

Solution

1. Step 1: Understand the role of document loaders

   Document loaders bring data into Langchain, preparing it for processing.

2. Step 2: Identify the purpose of custom loaders

   Custom loaders allow loading from any source and preprocessing like cleaning or adding metadata.

3. Final Answer:

   To load and preprocess documents from any data source before use -> Option A

4. Quick Check:

   Custom loaders = load + preprocess documents [OK]

Hint: Custom loaders bring in and prep data from any source [OK]

Common Mistakes:

• Confusing loaders with UI components
• Thinking loaders train models
• Assuming loaders manage databases

2. Which of the following is the correct way to start defining a custom document loader class in Langchain?

easy

A. class MyLoader():\n def load(self):\n pass

B. def MyLoader():\n return BaseLoader.load()

C. class MyLoader(BaseLoader):\n def load(self):\n pass

D. class MyLoader(BaseLoader):\n def read(self):\n pass

Solution

1. Step 1: Check inheritance from BaseLoader

   Custom loaders must inherit from BaseLoader to fit Langchain's pattern.

2. Step 2: Confirm the method name is load

   The required method to implement is load(), not read().

3. Final Answer:

   class MyLoader(BaseLoader):\n def load(self):\n pass -> Option C

4. Quick Check:

   Inherit BaseLoader + define load() method [OK]

Hint: Inherit BaseLoader and implement load() method [OK]

Common Mistakes:

• Not inheriting from BaseLoader
• Using wrong method name like read()
• Defining function instead of class

3. Given this custom loader code snippet, what will print(docs) output?

from langchain.schema import Document
from langchain.document_loaders.base import BaseLoader

class SimpleLoader(BaseLoader):
    def load(self):
        return [Document(page_content='Hello World', metadata={'source': 'test'})]

loader = SimpleLoader()
docs = loader.load()

medium

A. [Document(page_content='Hello World', metadata={'source': 'test'})]

B. ['Hello World']

C. None

D. SyntaxError

Solution

1. Step 1: Analyze the load method return value

   The load method returns a list with one Document object containing content and metadata.

2. Step 2: Understand what docs holds

   docs is assigned the list returned, so printing docs shows the list with the Document instance.

3. Final Answer:

   [Document(page_content='Hello World', metadata={'source': 'test'})] -> Option A

4. Quick Check:

   load() returns list of Document objects [OK]

Hint: load() returns list of Document objects, not strings [OK]

Common Mistakes:

• Expecting a list of strings instead of Document objects
• Thinking load() returns None
• Assuming syntax error due to imports

4. Identify the error in this custom loader code:

from langchain.document_loaders.base import BaseLoader

class FaultyLoader(BaseLoader):
    def load(self):
        docs = []
        docs.append('Just a string')
        return docs

loader = FaultyLoader()
loader.load()

medium

A. The load method must be named read

B. BaseLoader cannot be inherited directly

C. The class must not have a load method

D. The load method should return Document objects, not strings

Solution

1. Step 1: Check what load returns

   The load method returns a list with a string instead of Document objects.

2. Step 2: Recall Langchain expects Document objects

   Custom loaders must return Document instances, not plain strings, for proper processing.

3. Final Answer:

   The load method should return Document objects, not strings -> Option D

4. Quick Check:

   load() must return Document objects [OK]

Hint: load() must return Document objects, not plain strings [OK]

Common Mistakes:

• Returning strings instead of Document objects
• Misnaming the load method
• Thinking BaseLoader can't be inherited

5. You want to create a custom loader that reads text files from a folder, adds the filename as metadata, and returns Document objects. Which approach correctly implements this?

hard

A. Use BaseLoader directly without subclassing, call load() with folder path argument

B. Inherit BaseLoader, implement load() to read files, create Document with content and metadata filename, return list

C. Create a function that returns file contents as strings without Document wrapping

D. Subclass BaseLoader but return raw file paths as strings in load()

Solution

1. Step 1: Confirm inheritance and method implementation

   Custom loaders must subclass BaseLoader and implement load() method.

2. Step 2: Ensure load reads files and returns Document objects with metadata

   Reading files and wrapping content in Document with filename metadata is correct.

3. Final Answer:

   Inherit BaseLoader, implement load() to read files, create Document with content and metadata filename, return list -> Option B

4. Quick Check:

   Subclass + load() returns Document list with metadata [OK]

Hint: Subclass BaseLoader, load files, wrap in Document with metadata [OK]

Common Mistakes:

• Not subclassing BaseLoader
• Returning strings instead of Document objects
• Ignoring metadata addition