Custom document loaders in LangChain - Mini Project: Build & Apply

Create a Custom Document Loader in Langchain

📖 Scenario: You are building a small app that reads text files from a folder and loads their content for further processing. Langchain does not have a built-in loader for your file type, so you will create a custom document loader class.

🎯 Goal: Build a custom document loader class in Langchain that reads all text files from a folder and returns their content as documents.

📋 What You'll Learn

Create a class called CustomTextLoader that inherits from BaseLoader.

Add an __init__ method that takes a folder_path string parameter.

Implement a load method that reads all .txt files in the folder and returns a list of documents with their content.

Use the Document class from Langchain to create document objects.

💡 Why This Matters

🌍 Real World

Custom document loaders let you bring your own data files into Langchain pipelines for AI processing, search, or analysis.

💼 Career

Many AI and data jobs require integrating custom data sources. Knowing how to build loaders helps you prepare data for language models and other tools.

1

Set up the folder path variable

Create a variable called folder_path and set it to the string './texts'.

LangChain

# Your code here

Need a hint?

Use a simple string assignment to set the folder path.

2

Create the CustomTextLoader class with __init__

Define a class called CustomTextLoader that inherits from BaseLoader. Add an __init__ method that takes self and folder_path as parameters and assigns folder_path to self.folder_path.

LangChain

folder_path = './texts'

# Define CustomTextLoader class here
# Your code here

Need a hint?

Remember to import BaseLoader from Langchain before defining the class.

3

Implement the load method to read .txt files

Inside CustomTextLoader, add a method called load that takes only self. Use os.listdir(self.folder_path) to get all files. For each file ending with .txt, open and read its content. Create a Document object with the content and add it to a list called documents. Return documents at the end.

LangChain

folder_path = './texts'

from langchain.document_loaders.base import BaseLoader
import os
from langchain.schema import Document

class CustomTextLoader(BaseLoader):
    def __init__(self, folder_path: str):
        self.folder_path = folder_path

    # Add load method here
    # Your code here

Need a hint?

Use os.path.join to build the full file path. Remember to import os and Document.

4

Instantiate and use the CustomTextLoader

Create an instance of CustomTextLoader called loader using the folder_path variable. Then call loader.load() and assign the result to a variable called docs.

LangChain

folder_path = './texts'

from langchain.document_loaders.base import BaseLoader
import os
from langchain.schema import Document

class CustomTextLoader(BaseLoader):
    def __init__(self, folder_path: str):
        self.folder_path = folder_path

    def load(self):
        documents = []
        for filename in os.listdir(self.folder_path):
            if filename.endswith('.txt'):
                path = os.path.join(self.folder_path, filename)
                with open(path, 'r', encoding='utf-8') as file:
                    content = file.read()
                documents.append(Document(page_content=content))
        return documents

# Create loader instance and load documents
# Your code here

Need a hint?

Instantiate the class with folder_path and call load() to get documents.