Code-aware text splitting in LangChain - Mini Project: Build & Apply

Code-aware Text Splitting with Langchain

📖 Scenario: You are building a tool to split a long text document that contains code snippets and regular text. You want to split the text so that code blocks stay intact and are not broken in the middle.

🎯 Goal: Create a Python script using Langchain's CodeAwareTextSplitter to split a given text into chunks without breaking code blocks.

📋 What You'll Learn

Create a variable text with a sample string containing both code blocks and normal text.

Create a CodeAwareTextSplitter instance with a chunk size of 50 characters.

Use the splitter's split_text method on the text variable.

Store the result in a variable called chunks.

💡 Why This Matters

🌍 Real World

This technique is useful when processing documents or notes that mix code and text, such as programming tutorials or technical documentation, ensuring code snippets remain whole.

💼 Career

Developers and data scientists often need to preprocess mixed content for tasks like summarization, search indexing, or chatbot input, making code-aware splitting a valuable skill.

1

Create the text variable with code and normal text

Create a variable called text and assign it this exact string including newlines and code blocks: """Here is some introduction text. ```python def greet(): print('Hello, world!') ``` This is the conclusion."""

LangChain

# Create the variable text with the given string
# Your code here

Need a hint?

Use triple quotes """ to create a multi-line string exactly as shown.

2

Create a CodeAwareTextSplitter instance

Import CodeAwareTextSplitter from langchain.text_splitter. Then create a variable called splitter and assign it a CodeAwareTextSplitter instance with chunk_size=50.

LangChain

text = """Here is some introduction text.

```python
def greet():
    print('Hello, world!')
```

This is the conclusion."""

# Import CodeAwareTextSplitter and create splitter with chunk_size=50
# Your code here

Need a hint?

Use the exact import statement and create the splitter with the parameter chunk_size=50.

3

Split the text using the splitter

Use the split_text method of the splitter variable to split the text variable. Assign the result to a variable called chunks.

LangChain

from langchain.text_splitter import CodeAwareTextSplitter

text = """Here is some introduction text.

```python
def greet():
    print('Hello, world!')
```

This is the conclusion."""

splitter = CodeAwareTextSplitter(chunk_size=50)

# Use splitter.split_text on text and assign to chunks
# Your code here

Need a hint?

Call split_text on the splitter object passing text as argument.

4

Complete by adding a comment about the chunks variable

Add a comment line that says exactly: # chunks now contains the split text preserving code blocks below the chunks assignment.

LangChain

from langchain.text_splitter import CodeAwareTextSplitter

text = """Here is some introduction text.

```python
def greet():
    print('Hello, world!')
```

This is the conclusion."""

splitter = CodeAwareTextSplitter(chunk_size=50)

chunks = splitter.split_text(text)

# Add a comment about chunks preserving code blocks
# Your code here

Need a hint?

Just add the exact comment line below the chunks assignment.