The Chunker Deserved Its Own Article Three Years Ago
Every retrieval article we have written names chunking as the biggest lever and then moves on to something more interesting. That is a fair description of how the industry treats it, and it is the component where our measured gains have been largest.
Why It Is Neglected
Because it is boring and because it looks solved. Every framework ships a splitter with a size and an overlap, both configurable, and the defaults produce something that works well enough to get a demonstration running.
Then the project ends up tuning the reranker, because that is the component with interesting choices. Our measurements say the ordering should be the reverse: on six projects where we measured both, chunking changes accounted for more improvement than every reranker change combined.
What We Do
Structure before size. We split on the document's own boundaries, headings and sections and list items, and only apply a size limit within those. A chunk that ends mid-sentence because a character count ran out is a chunk that will retrieve for the wrong question.
One idea per chunk, headings carried into the text so a passage says what it is about, and tables never split. That last rule sounds narrow and prevents a whole class of the failures we described when writing about pages that are pictures.
The Direction the Research Points
Koutsiaris published work in 2026 on chunking driven by predicted information need, segmenting a document according to what someone is likely to want from it rather than according to its length.
That is the principle we had arrived at without stating it. Our structure-first rule works because document structure usually correlates with information need: an author put a heading there because a reader would look for that thing. Where the structure is poor, the correlation breaks and our chunker does badly.
| Rule | Why |
|---|---|
| Split on structure, then limit size | A boundary means something to a reader |
| Carry headings into the chunk text | A passage should say what it is about |
| Never split a table or a list | The relation is the content |
| Overlap only across a real boundary | Duplication that buys nothing costs index |
The First Thing We Got Wrong
Large overlap, for years, because it is the safe default and nobody revisits a safe default. When we finally measured, cutting the overlap removed nearly a fifth of the vectors in one corpus with no measurable quality change.
Overlap exists to stop an answer being split across a boundary. Where the boundaries are structural, that rarely happens, and the overlap is buying insurance against a failure the structure has already prevented. We now use overlap only where a section genuinely continues across a page break.
The Second Thing We Got Wrong
A single chunk size for a mixed corpus. A specification table, a procedure and a page of narrative prose have different natural sizes, and one setting serves the average of a distribution that has no meaningful average.
Chunk size is now a property of the document type rather than of the pipeline. That is more configuration, it required classifying incoming documents, and it produced the second largest single improvement we have measured after the move to structure-first splitting.
How We Tune It
Against the labelled set, one change at a time, three runs each. Chunking changes require a full re-index, so the loop is slow and that slowness is why people tune the reranker instead, which is fast and matters less.
We budget two days for chunking work at the start of a project and treat it as the first optimisation rather than the last. On every project where we did that, the later optimisations mattered less than expected, which is the correct order to discover that in.
The Setting Nobody Should Copy
Ours. The numbers that work for a technical manual corpus in German with heavy tables are not the numbers for a knowledge base of short English articles, and publishing a chunk size as advice is how a default becomes an industry habit.
What transfers is the order of operations: structure first, size within structure, per document type, measured on your own labelled set. The numbers that come out of that are yours and they are not interesting to anyone else.
What We Do Not Claim
We do not claim structure-first always wins. On documents with no reliable structure, scanned material or poorly converted files, it degrades to size-based splitting and we have nothing better to offer than fixing the conversion first.
We also do not claim chunking is the largest lever for everyone. It is the largest we have measured, on our corpora, which are document-heavy and structurally rich. A corpus of short records would have most of its quality decided elsewhere.