Metadata preservation during splitting in LangChain - Step-by-Step Execution

Concept Flow - Metadata preservation during splitting

Input Document with Metadata

↓

Split Document into Chunks

↓

Copy Metadata to Each Chunk

↓

Output: List of Chunks with Metadata

The document is split into smaller parts, and metadata is copied to each part to keep context.

Execution Sample

LangChain

doc = Document(page_content='Hello world', metadata={'source': 'file1'})
splitter = CharacterTextSplitter(chunk_size=5)
chunks = splitter.split_documents([doc])

Splits a document into chunks while keeping the original metadata on each chunk.

Execution Table

Step	Action	Input Document	Chunks Created	Metadata on Chunks
1	Start with one document	{content: 'Hello world', metadata: {'source': 'file1'}}	0	N/A
2	Split content into chunks of size 5	Content split into ['Hello', ' world']	2	N/A
3	Assign metadata to each chunk	Each chunk gets metadata {'source': 'file1'}	2	All chunks have {'source': 'file1'}
4	Output chunks with metadata	Chunks ready for use	2	Metadata preserved on all chunks

💡 All chunks created and metadata copied, splitting complete

Variable Tracker

Variable	Start	After Split	Final
doc	{content: 'Hello world', metadata: {'source': 'file1'}}	Same	Same
chunks	[]	[{content: 'Hello'}, {content: ' world'}]	[{content: 'Hello', metadata: {'source': 'file1'}}, {content: ' world', metadata: {'source': 'file1'}}]

Key Moments - 2 Insights

Why does each chunk need its own copy of metadata?

Because after splitting, each chunk is treated as a separate document. The metadata must be copied to keep the original context, as shown in step 3 of the execution_table.

Does splitting change the original document's metadata?

No, the original document's metadata stays the same. The metadata is copied to new chunk objects, not removed or altered from the original, as seen in variable_tracker.

Visual Quiz - 3 Questions

Look at the execution_table at step 2. How many chunks are created after splitting?

A 1

B 2

C 3

D 0

💡 Hint

Check the 'Chunks Created' column at step 2 in the execution_table.

According to variable_tracker, what is the metadata of the final chunks?

A Empty dictionary {}

B No metadata

C {'source': 'file1'}

D Metadata removed

💡 Hint

Look at the 'chunks' row under the 'Final' column in variable_tracker.

If the chunk size was increased to 11, how would the number of chunks change?

A Fewer chunks than before

B More chunks than before

C Same number of chunks

D No chunks created

💡 Hint

Refer to how chunk_size affects splitting in the execution_sample code.

Concept Snapshot

Metadata preservation during splitting:
- Input document has content + metadata
- Split content into smaller chunks
- Copy metadata to each chunk
- Output chunks keep original metadata
- Keeps context for each piece after splitting

Full Transcript

This visual execution shows how a document with metadata is split into smaller chunks while keeping the metadata on each chunk. First, the document content is split into parts based on chunk size. Then, the metadata from the original document is copied to each chunk so that context is preserved. The execution table traces each step from starting with one document, splitting content, assigning metadata, to outputting chunks with metadata. The variable tracker shows how the chunks array changes from empty to holding chunks with metadata. Key moments clarify why metadata copying is necessary and that the original document's metadata remains unchanged. The quiz tests understanding of chunk count, metadata presence, and chunk size effects. This helps beginners see how metadata preservation works during document splitting in Langchain.

Practice

1. What is the main benefit of preserving metadata when splitting documents in Langchain?

easy

A. It speeds up the splitting process.

B. It removes unnecessary text from documents.

C. It reduces the size of each chunk.

D. It keeps important context linked to each chunk.

Solution

1. Step 1: Understand what metadata means in this context

   Metadata is extra information about the document, like source or tags, that helps keep context.

2. Step 2: Recognize why preserving metadata is useful

   When splitting, keeping metadata ensures each chunk still knows where it came from and what it relates to.

3. Final Answer:

   It keeps important context linked to each chunk. -> Option D

4. Quick Check:

   Metadata preservation = keeps context [OK]

Hint: Metadata keeps context linked to chunks [OK]

Common Mistakes:

• Thinking metadata speeds splitting
• Confusing metadata with chunk size
• Believing metadata removes text

2. Which method in Langchain is used to split documents while preserving metadata?

easy

A. split_documents

B. preserve_metadata

C. chunk_text

D. split_text

Solution

1. Step 1: Recall the method name for splitting with metadata

   The method designed to split documents and keep metadata is called split_documents.

2. Step 2: Eliminate incorrect options

   Other options like preserve_metadata, chunk_text, and split_text are not valid Langchain methods for this purpose.

3. Final Answer:

   split_documents -> Option A

4. Quick Check:

   Method to split with metadata = split_documents [OK]

Hint: Remember method name: split_documents [OK]

Common Mistakes:

• Using split_text instead of split_documents
• Assuming preserve_metadata is a method
• Confusing chunk_text with splitting

3. Given this code snippet, what will be the metadata of the first chunk after splitting?

docs = [Document(page_content='Hello world', metadata={'source': 'file1'})]
splitter = RecursiveCharacterTextSplitter(chunk_size=5, chunk_overlap=0)
chunks = splitter.split_documents(docs)
print(chunks[0].metadata)

medium

A. null

B. {}

C. {'source': 'file1'}

D. {'chunk': 0}

Solution

1. Step 1: Understand the input document metadata

   The original document has metadata {'source': 'file1'}.

2. Step 2: Check if splitting preserves metadata

   The split_documents method keeps metadata for each chunk, so the first chunk keeps {'source': 'file1'}.

3. Final Answer:

   {'source': 'file1'} -> Option C

4. Quick Check:

   Metadata preserved = {'source': 'file1'} [OK]

Hint: Chunks keep original metadata unchanged [OK]

Common Mistakes:

• Expecting empty metadata after split
• Assuming metadata is lost or null
• Thinking chunk index is added to metadata

4. What is wrong with this code if the metadata is missing after splitting?

docs = [Document(page_content='Test text', metadata={'id': 123})]
splitter = RecursiveCharacterTextSplitter(chunk_size=4, chunk_overlap=1)
chunks = splitter.split_texts(docs)
print(chunks[0].metadata)

medium

A. Using split_texts instead of split_documents loses metadata.

B. Chunk size is too small to keep metadata.

C. Metadata key 'id' is invalid and removed.

D. Overlap must be zero to preserve metadata.

Solution

1. Step 1: Identify the method used for splitting

   The code uses split_texts which only splits text strings, not Document objects with metadata.

2. Step 2: Understand consequence on metadata

   Since split_texts works on strings, metadata is lost. The correct method to preserve metadata is split_documents.

3. Final Answer:

   Using split_texts instead of split_documents loses metadata. -> Option A

4. Quick Check:

   split_texts loses metadata [OK]

Hint: Use split_documents to keep metadata [OK]

Common Mistakes:

• Thinking chunk size affects metadata
• Believing metadata keys cause loss
• Assuming overlap affects metadata

5. You want to split a document into chunks of 100 characters with 20 characters overlapping, preserving metadata. Which code correctly achieves this?

hard

A. splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20)
chunks = splitter.split_texts(docs)

B. splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20)
chunks = splitter.split_documents(docs)

C. splitter = RecursiveCharacterTextSplitter(chunk_size=80, chunk_overlap=20)
chunks = splitter.split_documents(docs)

D. splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=0)
chunks = splitter.split_documents(docs)

Solution

1. Step 1: Set chunk size and overlap correctly

   The question asks for 100 characters chunk size and 20 characters overlap, so chunk_size=100 and chunk_overlap=20.

2. Step 2: Use the correct splitting method to preserve metadata

   To keep metadata, use split_documents method, not split_texts.

3. Final Answer:

   splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20) chunks = splitter.split_documents(docs) -> Option B

4. Quick Check:

   Correct chunk size, overlap, and method = splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20) chunks = splitter.split_documents(docs) [OK]

Hint: Match chunk size, overlap, and use split_documents [OK]

Common Mistakes:

• Using split_texts instead of split_documents
• Setting wrong chunk size or overlap
• Using zero overlap when asked for 20