Teams building RAG systems spend most of their time on model selection and vector database configuration. The part that usually has the biggest impact on retrieval accuracy gets five minutes of attention: how documents are chunked.
This isn't a minor implementation detail. We've seen chunking changes improve retrieval precision by 30–40% on the same corpus, same model, same embedding approach. Get it wrong and you're retrieving half-answers, split context, or noise that confuses the LLM downstream.
Why chunking matters
When a user asks a question, the retrieval system embeds that query and finds the chunks with the most similar embeddings. If the relevant information is split across two chunks, or buried in a chunk that also contains unrelated content, the right answer may score poorly compared to a noisier but more topically compact chunk.
The goal of chunking isn't to split documents evenly. It's to create the smallest unit that still contains self-contained, coherent context relevant to likely queries.
Fixed-size chunking
The simplest approach: split every document into chunks of N tokens (or characters), with optional overlap.
# 500 tokens, 50-token overlap
chunks = [text[i:i+500] for i in range(0, len(text), 450)]
When it works: preprocessed content that's already uniform — FAQs written in consistent format, product catalogs with identical record structures, API documentation with predictable section lengths.
When it fails: narrative documents, contracts, technical guides, or anything where a single concept spans a paragraph. Fixed chunking splits sentences mid-thought, separates a header from its content, and creates chunks that are syntactically coherent but semantically incomplete.
The overlap is a partial fix: it ensures the boundary content appears in two adjacent chunks, reducing hard splits. But it doesn't solve the fundamental issue that fixed-size chunks are content-agnostic.
Semantic chunking
Semantic chunking splits on content boundaries rather than character counts. The two main variants:
Sentence-boundary chunking: group sentences into chunks based on embedding similarity. Start a new chunk when the cosine distance between adjacent sentences exceeds a threshold. More computationally expensive at index time, but produces chunks that actually correspond to coherent topics.
Paragraph-boundary chunking: treat each paragraph as a natural unit, then merge short paragraphs and split very long ones. Works well for content that was written with paragraph structure in mind — most business documents, blog posts, and support articles.
| Approach | Index time cost | Retrieval precision | Good for | |---|---|---|---| | Fixed-size | Very low | Low-medium | Uniform, short-form content | | Sentence boundary | Medium | Medium-high | Narrative docs, long articles | | Paragraph boundary | Low | Medium-high | Business docs, support content | | Hierarchical | Medium-high | High | Technical docs, contracts, manuals |
Hierarchical chunking
Hierarchical chunking stores each document at multiple granularities: parent chunks (larger, more context) and child chunks (smaller, more precise).
The retrieval pattern changes: you embed and search at the child level (small chunks match queries precisely), but you retrieve and inject the parent chunk into the LLM context (which provides enough surrounding context for the answer to make sense).
Document
├── Section 1 (parent chunk)
│ ├── Paragraph 1.1 (child chunk)
│ ├── Paragraph 1.2 (child chunk)
│ └── Paragraph 1.3 (child chunk)
├── Section 2 (parent chunk)
│ ├── ...
This approach consistently outperforms flat chunking for technical documentation, contracts, and long-form reference material — anything where questions require context from surrounding paragraphs to answer correctly.
The tradeoff is indexing complexity: you're maintaining two chunk hierarchies and need logic to map child matches back to parent chunks. Worth it for document types where precision matters; overkill for simple FAQs.
Overlap strategies
Whatever chunking approach you use, overlap is almost always worth including. The question is how much.
Too little overlap (0–5%): hard boundaries create retrieval gaps. Sentences at the edge of chunks may score poorly because their context is in an adjacent chunk.
Too much overlap (30%+): chunks become redundant. When multiple chunks contain nearly identical content, you're burning vector index capacity and potentially injecting duplicate context into the LLM.
The practical range for most content: 10–15% overlap. For long-context technical documents where continuity is critical: up to 20%.
Chunk size vs retrieval precision
Smaller chunks are more precise but lose context. Larger chunks capture more context but match fewer specific queries accurately.
A concrete tradeoff:
- 512-token chunks: high precision on specific factual queries, poor performance on questions requiring synthesis across a section
- 1024-token chunks: lower precision on narrow queries, better performance on questions needing broader context
- 2048-token chunks: good for contract review or technical spec queries; poor for exact-lookup tasks like "what is the refund window?"
The right chunk size depends on your query distribution. Before picking a number, look at 50–100 example queries from your actual use case and ask: are these narrow lookups or broad synthesis questions?
How to test your chunking strategy
Don't guess. Run a structured eval:
- Take 50–100 representative queries from your use case
- For each query, identify the ground-truth document passage that contains the correct answer
- Run retrieval with your chunking strategy, get the top-5 chunks
- Score: was the ground-truth passage represented in the top-5?
This is recall@5. A well-tuned chunking strategy on business documents should hit 80–90% recall@5. If you're at 60–70%, the chunking strategy is usually the first thing to change before touching embedding models or retrieval parameters.
The chunking decision tree
Start here when evaluating a new corpus for a RAG development project:
- Is the content uniform (all FAQs, all product records)? → fixed-size with small overlap
- Is it narrative/prose without consistent paragraph structure? → sentence-boundary semantic chunking
- Is it well-structured business content (support docs, policies, wiki pages)? → paragraph-boundary chunking
- Is it technical documentation, contracts, or long-form reference material? → hierarchical chunking
- Is retrieval precision critical (healthcare, legal, financial use cases)? → hierarchical regardless of content type
Two implementation notes that often get skipped:
Strip boilerplate before chunking. Headers, footers, navigation elements, and repeated disclaimers pollute embeddings and reduce retrieval quality. Clean the document first.
Chunk metadata is as important as the chunk content. Store document title, section header, page number, and source URL with every chunk. The LLM needs this to generate grounded, citable answers.