Metadata preservation during splitting in LangChain

Introduction

When you split a large document into smaller parts, you want to keep important extra information (metadata) with each part. This helps you remember details like the source or author for each piece.

You have a long text and want to break it into smaller chunks for easier processing.

You want to keep track of where each chunk came from, like the original file name or page number.

You need to search or analyze parts of a document but still want to know their context.

You want to store document pieces in a database with their related info.

You are building a chatbot that answers questions using parts of documents and need to show source info.

Syntax

LangChain

from langchain.text_splitter import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
docs = text_splitter.split_documents(documents)

split_documents takes a list of documents with metadata and returns smaller documents keeping the metadata.

You can customize chunk_size and chunk_overlap to control how big each piece is and how much they share.

Examples

This splits a short text into chunks of 5 characters without overlap, keeping the source metadata.

LangChain

from langchain.schema import Document
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Original document with metadata
original_doc = Document(page_content="Hello world! This is a test.", metadata={"source": "file1.txt"})

# Splitter setup
splitter = RecursiveCharacterTextSplitter(chunk_size=5, chunk_overlap=0)

# Split document
chunks = splitter.split_documents([original_doc])

Chunks overlap by 2 characters and keep the author info with each chunk.

LangChain

from langchain.schema import Document
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Document with author metadata
doc = Document(page_content="This is a longer text that will be split.", metadata={"author": "Alice"})

splitter = RecursiveCharacterTextSplitter(chunk_size=10, chunk_overlap=2)
chunks = splitter.split_documents([doc])

Sample Program

This program splits a document into smaller parts, each keeping the original metadata like source and page number. It prints each chunk's text and metadata.

LangChain

from langchain.schema import Document
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Create a document with metadata
text = "Langchain helps you build apps with language models."
metadata = {"source": "example.txt", "page": 1}
doc = Document(page_content=text, metadata=metadata)

# Setup splitter
splitter = RecursiveCharacterTextSplitter(chunk_size=20, chunk_overlap=5)

# Split document
chunks = splitter.split_documents([doc])

# Print each chunk's content and metadata
for i, chunk in enumerate(chunks, 1):
    print(f"Chunk {i} content: {chunk.page_content}")
    print(f"Chunk {i} metadata: {chunk.metadata}\n")

Important Notes

Always check that metadata is preserved after splitting to avoid losing important info.

Metadata can include anything like source, author, date, or tags.

Adjust chunk size and overlap to balance between chunk length and context preservation.

Summary

Splitting documents keeps metadata so you don't lose context.

Use split_documents method to split while preserving metadata.

Customize chunk size and overlap to fit your needs.

Practice

1. What is the main benefit of preserving metadata when splitting documents in Langchain?

easy

A. It speeds up the splitting process.

B. It removes unnecessary text from documents.

C. It reduces the size of each chunk.

D. It keeps important context linked to each chunk.

Solution

1. Step 1: Understand what metadata means in this context

   Metadata is extra information about the document, like source or tags, that helps keep context.

2. Step 2: Recognize why preserving metadata is useful

   When splitting, keeping metadata ensures each chunk still knows where it came from and what it relates to.

3. Final Answer:

   It keeps important context linked to each chunk. -> Option D

4. Quick Check:

   Metadata preservation = keeps context [OK]

Hint: Metadata keeps context linked to chunks [OK]

Common Mistakes:

• Thinking metadata speeds splitting
• Confusing metadata with chunk size
• Believing metadata removes text

2. Which method in Langchain is used to split documents while preserving metadata?

easy

A. split_documents

B. preserve_metadata

C. chunk_text

D. split_text

Solution

1. Step 1: Recall the method name for splitting with metadata

   The method designed to split documents and keep metadata is called split_documents.

2. Step 2: Eliminate incorrect options

   Other options like preserve_metadata, chunk_text, and split_text are not valid Langchain methods for this purpose.

3. Final Answer:

   split_documents -> Option A

4. Quick Check:

   Method to split with metadata = split_documents [OK]

Hint: Remember method name: split_documents [OK]

Common Mistakes:

• Using split_text instead of split_documents
• Assuming preserve_metadata is a method
• Confusing chunk_text with splitting

3. Given this code snippet, what will be the metadata of the first chunk after splitting?

docs = [Document(page_content='Hello world', metadata={'source': 'file1'})]
splitter = RecursiveCharacterTextSplitter(chunk_size=5, chunk_overlap=0)
chunks = splitter.split_documents(docs)
print(chunks[0].metadata)

medium

A. null

B. {}

C. {'source': 'file1'}

D. {'chunk': 0}

Solution

1. Step 1: Understand the input document metadata

   The original document has metadata {'source': 'file1'}.

2. Step 2: Check if splitting preserves metadata

   The split_documents method keeps metadata for each chunk, so the first chunk keeps {'source': 'file1'}.

3. Final Answer:

   {'source': 'file1'} -> Option C

4. Quick Check:

   Metadata preserved = {'source': 'file1'} [OK]

Hint: Chunks keep original metadata unchanged [OK]

Common Mistakes:

• Expecting empty metadata after split
• Assuming metadata is lost or null
• Thinking chunk index is added to metadata

4. What is wrong with this code if the metadata is missing after splitting?

docs = [Document(page_content='Test text', metadata={'id': 123})]
splitter = RecursiveCharacterTextSplitter(chunk_size=4, chunk_overlap=1)
chunks = splitter.split_texts(docs)
print(chunks[0].metadata)

medium

A. Using split_texts instead of split_documents loses metadata.

B. Chunk size is too small to keep metadata.

C. Metadata key 'id' is invalid and removed.

D. Overlap must be zero to preserve metadata.

Solution

1. Step 1: Identify the method used for splitting

   The code uses split_texts which only splits text strings, not Document objects with metadata.

2. Step 2: Understand consequence on metadata

   Since split_texts works on strings, metadata is lost. The correct method to preserve metadata is split_documents.

3. Final Answer:

   Using split_texts instead of split_documents loses metadata. -> Option A

4. Quick Check:

   split_texts loses metadata [OK]

Hint: Use split_documents to keep metadata [OK]

Common Mistakes:

• Thinking chunk size affects metadata
• Believing metadata keys cause loss
• Assuming overlap affects metadata

5. You want to split a document into chunks of 100 characters with 20 characters overlapping, preserving metadata. Which code correctly achieves this?

hard

A. splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20)
chunks = splitter.split_texts(docs)

B. splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20)
chunks = splitter.split_documents(docs)

C. splitter = RecursiveCharacterTextSplitter(chunk_size=80, chunk_overlap=20)
chunks = splitter.split_documents(docs)

D. splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=0)
chunks = splitter.split_documents(docs)

Solution

1. Step 1: Set chunk size and overlap correctly

   The question asks for 100 characters chunk size and 20 characters overlap, so chunk_size=100 and chunk_overlap=20.

2. Step 2: Use the correct splitting method to preserve metadata

   To keep metadata, use split_documents method, not split_texts.

3. Final Answer:

   splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20) chunks = splitter.split_documents(docs) -> Option B

4. Quick Check:

   Correct chunk size, overlap, and method = splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20) chunks = splitter.split_documents(docs) [OK]

Hint: Match chunk size, overlap, and use split_documents [OK]

Common Mistakes:

• Using split_texts instead of split_documents
• Setting wrong chunk size or overlap
• Using zero overlap when asked for 20