Chunking Strategies for RAG: Size, Overlap, and Best Practices

What Is Chunking for RAG and Why Does It Matter?

Chunking for RAG is the process of breaking large documents into smaller, manageable pieces before converting them into embeddings for retrieval. It might sound like a minor preprocessing step, but it's actually one of the biggest levers you have to improve your RAG system's performance.

Here's the problem: when your retrieval system underperforms, most developers immediately blame the embedding model or the vector database. But the real issue is often hiding in plain sight. Even a perfect retrieval system fails if it searches over poorly prepared data.

If you're building RAG systems from scratch, understanding document splitting RAG fundamentals will save you countless hours of debugging. Your chunks need to accomplish two things simultaneously: they must be easy for vector search to find, and they must give the LLM enough context to generate useful answers.

NVIDIA tested seven chunking strategies across five datasets and found that performance varied significantly based on the approach. The difference between a good and bad chunking strategy can mean a 40% improvement in retrieval accuracy, according to recent benchmarks.

How Does Chunk Size Affect Retrieval Quality?

The optimal chunk size isn't a single magic number. It depends on your document types, query patterns, and what you're trying to accomplish.

Research from multiple sources points to a general sweet spot between 128 and 512 tokens for most use cases. But the specifics matter:

Smaller chunks (128 to 256 tokens) excel at precise, fact-based queries. When someone asks "What was the Q2 revenue for ACME Corp?", a small chunk containing just that figure is easier to retrieve and more relevant than a larger section discussing multiple quarters.

Larger chunks (512 to 1024 tokens) perform better for analytical or explanatory queries. Questions like "Explain the methodology behind this study" need more context to produce a coherent answer.

When converting text to searchable embeddings, remember that embedding models produce a single fixed-size vector regardless of input length. A 200-word chunk and a 2000-word chunk both become one vector. This means larger chunks can dilute specific information, while smaller chunks may lose important context.

Here's a practical breakdown by document type:

• FAQs: 128 to 256 tokens (matches question-answer pairs naturally)
• Technical docs: 300 to 500 tokens (preserves step-by-step procedures)
• Research papers: 512 to 1024 tokens (maintains complex arguments)
• Legal documents: 600 to 1000 tokens (keeps clause integrity)
• Code documentation: 200 to 400 tokens (aligns with function-level context)

What Are the Main Text Chunking Strategies?

There's no single best approach. Each strategy trades off context preservation against retrieval precision in different ways.

Fixed-Size Chunking

The simplest method: split text into uniform segments based on character, word, or token count. Fast to implement, predictable to manage. The downside? It has zero awareness of document structure. Sentences get cut mid-word, paragraphs break in awkward places, and related ideas end up scattered across chunks.

Use this as a starting point when you're prototyping or don't know your data well yet.

Recursive Character Splitting

This is the default choice for about 80% of RAG applications. It uses a hierarchy of separators to find natural boundaries, attempting splits at paragraph breaks first, then sentences, then spaces.

The method respects document structure while still producing reasonably consistent chunk sizes. LangChain's RecursiveCharacterTextSplitter is the most common implementation, with default separators of ["\n\n", "\n", " ", ""].

In Chroma's research, recursive splitting achieved 88 to 89% recall with 400-token chunks using text-embedding-3-large. That's a solid baseline for most projects.

Sentence-Based Chunking

Each chunk contains complete sentences. No thought gets cut off mid-expression. This works well when your queries align with sentence-level information, like customer support questions or conversational AI.

The tradeoff: sentence lengths vary wildly, leading to inconsistent chunk sizes. A single sentence might be 10 tokens or 200 tokens.

Semantic Chunking

Instead of splitting on structure, semantic chunking uses embedding similarity to group related content together. The process works like this:

1. Split text into sentences
2. Generate embeddings for each sentence
3. Compare similarity between consecutive sentences
4. Create chunk boundaries where similarity drops significantly

This approach can improve recall by 2 to 3 percentage points over recursive splitting, according to Chroma's benchmarks. LLM-based semantic chunking achieved the highest scores in multiple tests, with 0.919 recall in one study.

But there's a catch. Every sentence needs its own embedding. For a 10,000-word document, you might generate 200 to 300 embeddings just for the chunking step. That's expensive if you're using API-based embedding services.

Agentic Chunking

The newest approach: let an LLM analyze each document and decide where to split it. The model can understand semantic meaning, identify topic transitions, and respect content structure like section headings and step-by-step instructions.

It produces the highest-quality chunks but is also the slowest and most expensive method. You're making an LLM call for document segmentation before you even start the retrieval pipeline.

How Should You Handle Overlap Chunking?

Overlap is your insurance policy against boundary problems. When you create chunks that share some content with their neighbors, you reduce the risk of important information getting split across chunks.

NVIDIA tested 10%, 15%, and 20% overlap values and found 15% performed best on their FinanceBench dataset with 1024-token chunks. Industry practice generally recommends 10 to 20% overlap as a starting point.

For a 500-token chunk, that means 50 to 100 tokens of overlap with adjacent chunks.

Here's how to think about overlap:

More overlap means better context preservation but higher storage costs, more redundant retrieval results, and increased processing time. Each overlapped token appears in multiple chunks, so you're storing and processing the same information multiple times.

Less overlap is more efficient but risks splitting critical sentences or ideas across chunk boundaries.

When you're understanding LLM token limits, keep in mind that overlap affects your total token count. Heavy overlap can significantly increase your vector database size and retrieval costs.

What Is the Optimal Chunk Size for Different Query Types?

Your users' query patterns should drive your chunking decisions.

Factoid queries (specific facts, names, dates): 256 to 512 tokens work best. The chunk should contain the answer and minimal surrounding noise.

Analytical queries (explanations, comparisons, reasoning): 1024+ tokens or page-level chunking. The LLM needs broader context to synthesize a coherent response.

Mixed queries: Start with 400 to 512 tokens as a balanced middle ground.

NVIDIA's testing confirmed this pattern across multiple datasets. Page-level chunking won their evaluation with 0.648 accuracy, but token-based approaches performed better for specific query types.

If you're measuring your system's performance, understanding RAG evaluation metrics will help you identify which chunking strategy actually improves your retrieval quality.

What Advanced Techniques Improve RAG Retrieval?

Late Chunking

Traditional chunking embeds each chunk independently, which means each piece loses context from the rest of the document. Late chunking flips this: embed the entire document first, then segment the token-level embeddings into chunks afterward.

This preserves global context within each chunk's embedding without requiring extra LLM calls. Research shows late chunking works particularly well as documents approach 8,000 tokens in length.

The tradeoff: it requires embedding models that support long context windows, and you're processing more tokens upfront.

Contextual Retrieval

Anthropic introduced this technique in 2024. Before embedding each chunk, you use an LLM to generate a contextual summary that situates the chunk within the broader document.

A chunk that originally said "Revenue grew by 3% over the previous quarter" becomes "This chunk is from ACME Corp's Q2 2023 SEC filing. Revenue grew by 3% over the previous quarter."

In Anthropic's tests, this reduced retrieval failures by up to 67% when combined with improving retrieval with reranking. The contextual embeddings give the vector search more to work with, especially for queries about "which company" or "what time period."

The cost consideration: you're making an LLM call for every chunk. Prompt caching can reduce this by up to 90%, but it's still more expensive than basic chunking.

Hierarchical Chunking

Create multiple chunk sizes with parent-child relationships. Index a chapter, its sections, its paragraphs, and its sentences. At query time, search across all levels and use the results to navigate to the most relevant granularity.

This approach is ideal for very large, complex documents like textbooks or legal contracts. You can answer both high-level summary questions and highly specific detail questions from the same knowledge base.

LlamaIndex's HierarchicalNodeParser makes implementation straightforward if you're building in Python.

What Mistakes Should You Avoid When Chunking?

Based on production RAG systems, these are the most common chunking errors:

Using only default settings. Most developers stick with 512-token chunks without testing alternatives. That might be fine, or it might be costing you 20% retrieval accuracy. You won't know until you evaluate.

Ignoring document structure. Naive character splitting doesn't care if it breaks a sentence mid-word or separates a heading from its content. Use structure-aware methods whenever possible.

Zero overlap. Boundary misses are a real problem. If critical information falls exactly at a chunk boundary, you'll never retrieve it correctly.

Over-chunking everything. Not all documents need chunking. For short, focused content like FAQs or product descriptions, chunking can actually hurt performance. Documents under 200,000 tokens might work better stuffed directly into the prompt.

When you're maximizing LLM context utilization, consider whether chunking is even necessary for your document size.

No metadata. Track which document each chunk came from, its position in the document, version information, and any relevant tags. This enables filtering, deduplication, and better citation in your responses.

How Do You Choose the Right Chunking Strategy?

Start with this decision tree:

1. Is your document short and focused? (Under 500 tokens, single topic) → Don't chunk at all. Embed the whole thing.
2. Is your document well-structured? (Clear headers, sections, paragraphs) → Use recursive character splitting or document-based chunking that respects structure.
3. Are your queries fact-based or analytical? → Fact-based: smaller chunks (256 to 512 tokens). Analytical: larger chunks (512 to 1024 tokens).
4. Is retrieval quality critical and budget flexible? → Test semantic or LLM-based chunking.
5. Are you dealing with PDFs or visual documents? → Consider page-level chunking or specialized AI document analysis tools that respect layout.

For most applications, start with RecursiveCharacterTextSplitter at 400 to 512 tokens with 10 to 20% overlap. This provides a solid baseline that you can optimize from.

How Should You Test and Evaluate Chunking Strategies?

Don't guess. Measure.

1. Create a test dataset. 50 to 100 representative documents with 20 to 30 realistic queries. Include edge cases from your expected usage.
2. Define success metrics. Recall@k measures whether relevant chunks appear in your top k results. Precision measures how many retrieved chunks are actually useful. MRR (Mean Reciprocal Rank) shows how highly relevant results rank.
3. Test 2 to 3 strategies. At minimum, compare your current approach against one alternative. Recursive splitting vs semantic chunking, or 256-token vs 512-token chunks.
4. Evaluate with humans and LLMs. Automated metrics catch obvious problems. Human review catches things metrics miss, like whether the retrieved context actually enables good answers.
5. Monitor in production. The queries you designed for might not match real user behavior. Track retrieval performance over time and iterate.

Understanding the tokenization process in language models helps you reason about why certain chunk sizes perform differently across models.

What Tools and Libraries Should You Use?

LangChain offers multiple splitters out of the box: CharacterTextSplitter, RecursiveCharacterTextSplitter, TokenTextSplitter, and specialized splitters for Markdown, HTML, and code. It's the most flexible option for Python developers.

LlamaIndex provides SentenceSplitter, SemanticSplitterNodeParser, and HierarchicalNodeParser. Strong support for advanced techniques like hierarchical chunking.

Unstructured specializes in parsing complex documents (PDFs, DOCXs, images) and offers "smart chunking" that respects document structure automatically.

Pinecone, Weaviate, and Chroma all have documentation on chunking best practices specific to their vector databases.

Ready to find the right tools for your RAG pipeline? Browse our AI tools directory to explore document processing, embedding, and vector database options that fit your workflow.

Final Thoughts

Chunking isn't glamorous, but it's foundational. A well-tuned chunking strategy can improve retrieval accuracy by 40% compared to naive approaches. A poorly chosen strategy can make even the best embedding model and vector database underperform.

The key takeaways:

• Start with recursive splitting at 400 to 512 tokens and 10 to 20% overlap
• Match chunk size to query type: smaller for facts, larger for analysis
• Always test on your actual data and measure with your real queries
• Consider advanced techniques like contextual retrieval or late chunking when basic approaches plateau

If you're serious about RAG fundamentals and architecture, chunking is where you should spend your optimization effort first. Get this right, and everything downstream improves.