11 Sep 2023 AI Engineering 4 min read

Chunking Strategies for Document QA

Chunking determines what a document QA system can retrieve at all. We compare fixed-size and recursive splitting, quantify chunk size and overlap trade-offs against the context windows of 2023, and explain why metadata is mandatory. With verified numbers from Pinecone, OpenAI and the Lost in the Middle paper — plus an outlook from September 2023.

The First Decision in Every Document QA Pipeline

Document question answering in 2023 follows a standard pattern: split documents into chunks, embed each chunk, store the vectors, retrieve the top-k chunks for a query, and let a language model generate an answer from them. Retrieval-augmented generation (RAG) is the accepted name for this pipeline. Every step gets attention — except the first one.

Chunking is decided once, early, and usually by accepting a library default. That is a mistake. The chunk is the unit of retrieval: what was never split correctly can never be retrieved correctly. In our client projects, changing the chunking strategy has moved answer quality more than any switch of embedding model or LLM.

Documents are chunked, embedded and indexed — vectors plus keywords. 1/4

Why Chunking Dominates Retrieval Quality

An embedding compresses a chunk into a single vector — 1,536 dimensions for OpenAI's text-embedding-ada-002, released in December 2022. The vector represents the average meaning of the chunk. A chunk that mixes three topics produces a vector that represents none of them well. A chunk that ends mid-sentence embeds a fragment.

The retriever can only rank what the chunker produced. If an answer spans a chunk boundary, no similarity metric will surface it intact. Hard limits apply as well: ada-002 accepts at most 8,191 tokens per input. Chunking failures are silent — the pipeline returns plausible but incomplete context, and the LLM fills the gaps with fluent guesswork.

Fixed Size Splitting

Fixed-size splitting cuts text into windows of N characters or tokens, optionally with overlap. It is deterministic, fast, requires no parsing, and works on any input. LangChain's base TextSplitter defaults to 4,000 characters per chunk with 200 characters of overlap; LlamaIndex currently defaults to 1,024 tokens with an overlap of 20.

The weakness is obvious: fixed windows ignore structure. They cut through sentences, separate tables from their headers, and detach headings from the paragraphs they describe. Fixed-size splitting is a reasonable baseline for homogeneous prose. It is a poor choice for contracts, manuals, or anything where layout carries meaning.

Recursive Splitting Respects Document Structure

Recursive splitting tries a hierarchy of separators and only falls back to the next level when a chunk is still too large. LangChain's RecursiveCharacterTextSplitter uses ["\n\n", "\n", " ", ""] by default: paragraphs first, then lines, then words. The result is chunks that end at natural boundaries whenever the size budget allows it.

This is a heuristic, not semantics. The splitter does not know that two paragraphs belong to the same argument. But because paragraph breaks correlate strongly with topic shifts, recursive splitting is the recommended default for generic text. For Markdown, HTML, and source code, separator lists tuned to the format do measurably better.

Chunk Size and Overlap Trade-offs

Chunk size is a trade-off, not an optimization with a single correct answer. Small chunks embed precisely but lose surrounding context; large chunks preserve context but dilute the vector and fill the prompt faster. With GPT-3.5-turbo's 4,096-token window, instruction, question, and answer compete with retrieved chunks for the same budget — roughly 3,000 tokens remain for context in practice.

Overlap of 10 to 20 percent mitigates losses at chunk boundaries; it does not remove them, and it grows the index and the embedding bill. At ada-002's price of $0.0001 per 1,000 tokens (since June 2023) that is cheap, but index size and query latency grow too. Pinecone's April 2023 guide recommends testing sizes between 128 and 1,024 tokens against representative queries instead of trusting defaults. Liu et al. add a related warning: retrieving more than 20 documents improved reader accuracy by only about 1.5 percent for GPT-3.5-Turbo. More context is not automatically better.

Metadata Turns Chunks Into Evidence

A chunk without provenance is a dead end. Store with every vector at minimum: source document, section heading, page or position, and ingestion date. Vector databases such as Pinecone, Weaviate, and Qdrant support metadata filters at query time — turning 'search everything' into 'search only the 2023 contracts'.

Metadata also decouples the retrieval unit from the generation unit. Store small chunks for precise matching, keep a parent reference in the metadata, and hand the surrounding section to the LLM. This small-to-large pattern beats raw top-k retrieval — at the price of extra bookkeeping.

What Chunking Does Not Solve

Chunking cannot repair broken extraction. If the PDF parser scrambles reading order or drops table cells, every downstream chunk inherits the damage. Chunking also does not fix embedding mismatch: a general-purpose model ranks domain jargon poorly regardless of where the segment boundaries sit.

Above all, chunk-based retrieval answers local questions. 'What is the notice period in section 8?' works. 'Summarize this 300-page report' does not — no top-k set of chunks contains a global view. Such tasks need map-reduce summarization or similar techniques, not a better splitter.

Outlook From September 2023

Context windows are growing fast: Anthropic's Claude 2 accepts 100,000 tokens, and OpenAI offers a 16k variant of GPT-3.5-turbo and a 32k GPT-4. It is tempting to declare chunking obsolete — just paste in the whole document. The Lost in the Middle results (Liu et al., July 2023) argue otherwise: models use the beginning and end of long contexts far better than the middle, and GPT-3.5-Turbo's QA accuracy fell below its closed-book baseline of 56.1 percent once the relevant passage sat mid-context.

Our expectation for the coming year: chunking stays, but gets smarter. Semantic splitting via embeddings, layout-aware PDF parsing, and evaluation-driven chunk-size selection will replace hard-coded defaults. Retrieval will decouple further from chunk boundaries. The teams that treat chunking as a measured engineering decision — not a default — will ship the better QA systems.