Directory loader for bulk documents in LangChain - Performance & Optimization

Performance: Directory loader for bulk documents

MEDIUM IMPACT

This affects the initial page load speed and responsiveness by controlling how many documents are loaded and processed at once.

Loading many documents from a directory to process in bulk

LangChain

from langchain.document_loaders import DirectoryLoader
loader = DirectoryLoader('docs')
for doc in loader.lazy_load():
    process(doc)

Loads documents one by one lazily, allowing processing and rendering to start earlier.

📈 Performance Gainreduces blocking time, improves LCP by streaming documents

Loading many documents from a directory to process in bulk

LangChain

from langchain.document_loaders import DirectoryLoader
loader = DirectoryLoader('docs')
docs = loader.load()

Loads all documents at once, blocking the process until all files are read and parsed.

📉 Performance Costblocks rendering for hundreds of milliseconds or more depending on file count and size

Performance Comparison

Pattern	DOM Operations	Reflows	Paint Cost	Verdict
Load all documents at once	N/A (backend load)	Blocks main thread causing delayed reflows	High paint cost due to delayed content	[X] Bad
Lazy load documents one by one	N/A (backend load)	Minimal blocking, allows incremental reflows	Lower paint cost, faster content display	[OK] Good

Rendering Pipeline

Bulk loading all documents triggers a long blocking task before rendering can proceed. Lazy loading breaks this into smaller chunks, allowing the browser to interleave rendering and processing.

→Script Execution

→Layout

→Paint

⚠️ BottleneckScript Execution blocking main thread

Core Web Vital Affected

LCP

This affects the initial page load speed and responsiveness by controlling how many documents are loaded and processed at once.

Optimization Tips

1Avoid loading all documents at once to prevent blocking rendering.

2Use lazy loading or batch processing to improve initial load speed.

3Monitor main thread blocking time to optimize document loading.

Performance Quiz - 3 Questions

What is the main performance issue when loading all documents from a directory at once?

A It blocks the main thread causing delayed rendering

B It increases CSS selector complexity

C It causes layout shifts after rendering

D It reduces network requests

DevTools: Performance

How to check: Record a performance profile while loading documents; look for long scripting tasks blocking rendering.

What to look for: Long tasks in the main thread timeline indicate blocking; shorter tasks with gaps show better lazy loading.