Menu

LangChain chunk_overlap=200: 2,181 of 4,155 Chunks Got No Overlap
📰

LangChain chunk_overlap=200: 2,181 of 4,155 Chunks Got No Overlap

DEV Community: python·jidonglab·10 days ago
#DL7GwZI9
#devtopython#devto#dev#overlap#chunk_overlap
Reading 0:00
15s threshold

This is the first line of a chunk from my test index:

That 14.2% is the number that explains the 32% crash rate.

14.2% of what? The answer sits at the end of the previous chunk, and the two chunks share zero characters. I had set LangChain chunk_overlap=200 , the setting every RAG tutorial uses to stop exactly this.

So I stopped trusting the parameter and measured it. I ran RecursiveCharacterTextSplitter over 452 of my own blog posts (3.5M characters) and checked how much text each pair of neighboring chunks really shared. More than half shared nothing.

TL;DR

LangChain chunk_overlap is a maximum, not a guarantee. RecursiveCharacterTextSplitter only carries over whole pieces, and those pieces are usually paragraphs.

If the last paragraph of a chunk is longer than chunk_overlap , nothing carries over.

At chunk_size=1000, chunk_overlap=200 , 2,181 of 4,155 seams (52.5%) had 0 characters of overlap.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.