Directory loader for bulk documents in LangChain - Step-by-Step Execution

Concept Flow - Directory loader for bulk documents

Start

↓

Specify directory path

↓

Initialize DirectoryLoader

↓

Load all files in directory

↓

Read each document

↓

Store documents in list

↓

Return list of documents

↓

End

The loader starts by specifying a folder path, then reads all files inside, loads each document, and returns them as a list.

Execution Sample

LangChain

from langchain.document_loaders import DirectoryLoader
loader = DirectoryLoader('docs')
docs = loader.load()
print(len(docs))

This code loads all documents from the 'docs' folder and prints how many documents were loaded.

Execution Table

Step	Action	Files Found	Documents Loaded	Output
1	Initialize DirectoryLoader with 'docs'	N/A	0	Loader ready
2	Scan 'docs' directory	3 files found	0	Files listed
3	Read file 1	3 files found	1	Document 1 loaded
4	Read file 2	3 files found	2	Document 2 loaded
5	Read file 3	3 files found	3	Document 3 loaded
6	Return documents list	3 files found	3	List of 3 documents
7	Print length of docs	3 files found	3	Output: 3

💡 All files in directory processed and documents loaded

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	Final
loader	None	DirectoryLoader instance	DirectoryLoader instance	DirectoryLoader instance	DirectoryLoader instance	DirectoryLoader instance
files_found	None	3 files	3 files	3 files	3 files	3 files
docs	[]	[]	[doc1]	[doc1, doc2]	[doc1, doc2, doc3]	[doc1, doc2, doc3]

Key Moments - 2 Insights

Why does the loader return a list of documents instead of a single document?

Because the directory can contain multiple files, the loader reads each file and returns all documents as a list, as shown in execution_table steps 3 to 6.

What happens if the directory is empty?

The loader finds zero files, so the documents list remains empty. The process ends quickly with an empty list, similar to step 2 but with zero files.

Visual Quiz - 3 Questions

Look at the execution table, how many documents are loaded after step 4?

A 2

B 1

C 3

D 0

💡 Hint

Check the 'Documents Loaded' column at step 4 in the execution_table.

At which step does the loader finish reading all files?

A Step 3

B Step 5

C Step 6

D Step 7

💡 Hint

Look for the last 'Read file' action in the execution_table.

If the directory had 5 files instead of 3, what would change in the variable_tracker?

A loader variable would change to 5

B files_found would remain 3

C docs list would have 5 documents after final step

D docs list would be empty

💡 Hint

See how docs list grows with files_found in variable_tracker.

Concept Snapshot

DirectoryLoader loads all files from a folder.
Initialize with folder path.
Call load() to read all documents.
Returns a list of documents.
Useful for bulk document processing.

Full Transcript

This visual execution shows how DirectoryLoader from langchain works. First, you create a loader with the folder path. Then it scans the folder and finds all files. It reads each file one by one, loading documents into a list. Finally, it returns the list of all documents. The example code loads documents from a folder named 'docs' and prints how many documents were loaded. The execution table traces each step, showing files found and documents loaded. The variable tracker shows how the docs list grows as files are read. Key moments clarify why the loader returns a list and what happens if the folder is empty. The quiz tests understanding of document counts and process steps. This helps beginners see how bulk document loading works step-by-step.

Practice

1. What is the main purpose of the DirectoryLoader in Langchain?

easy

A. To convert documents into text format

B. To save documents to a directory

C. To load multiple documents from a folder at once

D. To create a new directory for documents

Solution

1. Step 1: Understand DirectoryLoader's role

   The DirectoryLoader is designed to read many files from a folder in one go.

2. Step 2: Compare options with this role

   Only To load multiple documents from a folder at once describes loading multiple documents from a folder, matching DirectoryLoader's purpose.

3. Final Answer:

   To load multiple documents from a folder at once -> Option C

4. Quick Check:

   DirectoryLoader loads many files = A [OK]

Hint: DirectoryLoader reads many files from a folder at once [OK]

Common Mistakes:

• Confusing loading with saving files
• Thinking it converts file formats
• Assuming it creates directories

2. Which of the following is the correct way to specify a glob pattern to load only text files using DirectoryLoader?

easy

A. glob='*.txt'

B. glob='text/*'

C. glob='**/*.doc'

D. glob='*.pdf'

Solution

1. Step 1: Understand glob pattern for text files

   The pattern '*.txt' matches all files ending with .txt in the directory.

2. Step 2: Evaluate each option

   glob='*.txt' correctly uses '*.txt' to select text files. Others select different or incorrect file types.

3. Final Answer:

   glob='*.txt' -> Option A

4. Quick Check:

   Use '*.txt' to pick text files [OK]

Hint: Use '*.txt' to select text files with glob [OK]

Common Mistakes:

• Using folder names instead of file patterns
• Using wrong file extensions
• Confusing recursive patterns with simple glob

3. Given the code snippet:

loader = DirectoryLoader('docs', glob='*.md')
docs = loader.load()

What will docs contain?

medium

A. A list of all markdown (.md) documents in the 'docs' folder

B. A single markdown document from the 'docs' folder

C. An error because glob pattern is incorrect

D. A list of all files in 'docs' regardless of extension

Solution

1. Step 1: Analyze DirectoryLoader usage

   The DirectoryLoader is set to load files from 'docs' folder with glob '*.md', meaning markdown files only.

2. Step 2: Understand load() output

   The load() method returns a list of documents matching the glob pattern, so docs will be a list of markdown files.

3. Final Answer:

   A list of all markdown (.md) documents in the 'docs' folder -> Option A

4. Quick Check:

   load() returns list of files matching glob [OK]

Hint: load() returns list of files matching glob pattern [OK]

Common Mistakes:

• Expecting a single document instead of a list
• Assuming glob pattern is ignored
• Thinking load() returns file paths instead of documents

4. What is wrong with this code snippet?

loader = DirectoryLoader('files', glob='*.txt')
docs = loader.load

medium

A. Incorrect glob pattern for text files

B. Missing parentheses after load method call

C. DirectoryLoader requires absolute path only

D. load method does not exist in DirectoryLoader

Solution

1. Step 1: Check method call syntax

   The load method must be called with parentheses: load()

2. Step 2: Verify other parts

   Glob pattern '*.txt' is correct, DirectoryLoader accepts relative paths, and load() method exists.

3. Final Answer:

   Missing parentheses after load method call -> Option B

4. Quick Check:

   Method calls need () to execute [OK]

Hint: Remember to add () when calling methods [OK]

Common Mistakes:

• Forgetting parentheses on method calls
• Assuming glob pattern is wrong
• Thinking DirectoryLoader needs absolute paths

5. You want to load all PDF and text files from a folder named 'resources' using DirectoryLoader. Which code snippet correctly achieves this?

hard

A. loader = DirectoryLoader('resources', glob='*.pdf;*.txt')

B. loader = DirectoryLoader('resources')
docs = loader.load() # loads all files

C. loader = DirectoryLoader('resources', glob='*.pdf|*.txt')
docs = loader.load()

D. loader = DirectoryLoader('resources', glob='*.pdf')
loader2 = DirectoryLoader('resources', glob='*.txt')
docs = loader.load() + loader2.load()

Solution

1. Step 1: Understand glob pattern limitations

   DirectoryLoader's glob does not support multiple patterns separated by ';' or '|'.

2. Step 2: Check each option

   loader = DirectoryLoader('resources', glob='*.pdf') loader2 = DirectoryLoader('resources', glob='*.txt') docs = loader.load() + loader2.load() loads PDFs and text files separately and combines results, which works correctly. loader = DirectoryLoader('resources') docs = loader.load() # loads all files loads all files but not filtered. Options A and C use invalid glob patterns.

3. Final Answer:

   loader = DirectoryLoader('resources', glob='*.pdf') loader2 = DirectoryLoader('resources', glob='*.txt') docs = loader.load() + loader2.load() -> Option D

4. Quick Check:

   Use separate loaders for multiple file types [OK]

Hint: Use separate loaders and combine for multiple file types [OK]

Common Mistakes:

• Trying to use multiple patterns in one glob string
• Assuming glob supports ';' or '|' separators
• Loading all files without filtering