RAG Chunking Strategies: Fixed, Structure-Aware & Semantic

Chunking turns a long source into retrieval units that an embedding model and search system can index. The boundary matters: a passage that is too broad may bring irrelevant material, while one that is too narrow may lose the heading, definition or steps that make it understandable. But there is no universally best chunk size—and “semantic” is not automatically better than a simple baseline.

Choose chunking as an experiment tied to your documents and questions. Preserve the structure people rely on, keep provenance with every passage, and compare retrieval and answer quality against a reproducible baseline before adding complexity.

What chunking changes in a RAG pipeline

A retriever ranks the units you indexed, not an abstract whole document. Chunking therefore affects what can be matched, how much text is returned, whether related information is split apart, how many embeddings must be stored, and how many near-duplicate passages compete for the model’s context. It also affects updates: a change to one section should not silently leave stale versions of its chunks searchable.

Start by asking what a useful evidence unit looks like for the task. A product question may need one specification and its heading. A troubleshooting question may need a complete procedure with prerequisites and warnings. A policy question may need the clause, its scope, and the section that defines the terms. Design for the evidence users need—not for an attractive chunk-size number.

Four useful chunking patterns

Fixed-size chunks: the baseline

Split text near a target token count, ideally at a sentence or paragraph boundary. This is easy to implement and repeat, making it a valuable baseline. It can still cut through a table, step sequence or definition, so preserve structural boundaries where possible and record the source heading with each piece.

Structure-aware recursive splitting

Prefer the document’s natural units—such as page, section, heading, paragraph, list or table—and split only when a unit exceeds the chosen budget. A practical implementation can walk this hierarchy from largest to smallest: keep a complete section when it fits; otherwise split it into paragraphs, then sentences. Avoid orphaning a bullet from its lead-in, a table row from its column labels, or a code sample from its explanation.

Semantic boundary detection

Some systems compare neighbouring sentences or sections and introduce a boundary when their meaning appears to shift. This can be worth testing for narrative material with weak headings, but it adds processing and tuning choices. In a NAACL 2025 evaluation, semantic chunking did not deliver consistent gains over simpler chunking across the tested retrieval and answer tasks; that is evidence against treating it as a default, not proof it can never help. Qu, Tu and Bao, “Is Semantic Chunking Worth the Computational Cost?”

Hierarchical or parent-child retrieval

Index smaller passages for precise matching while retaining a larger parent section for context. A system can retrieve a child passage and then include its heading, nearby text or parent section in the answer context. This may help when a precise fact depends on surrounding explanation, but it requires stable parent-child references and a limit on how much expanded text reaches the model.

Set size and overlap by measurement

Published experiments do not establish a universal token range. In a multi-dataset study, the chunk sizes associated with stronger retrieval varied with the task and embedding model. Other evaluations find that semantic or document-aware segmentation can help particular material. The useful conclusion is not “always use small chunks” or “always chunk semantically”; it is that the source format, questions, embedding model and evaluation set all matter.

Choose a manageable set of candidate sizes that fit your embedding model and downstream context budget, then test them on the same representative questions. Include short fact lookups and questions that require context across several sentences or sections. Keep overlap as a controlled variable: it can rescue boundary-spanning evidence, but also duplicates text, consumes storage and may crowd out distinct results. Begin with none or a modest amount, then keep it only if the evaluation demonstrates value.

Record the tokenizer and model used when selecting a token budget. The same character count can yield different token counts, and changing the embedding model can change which chunk boundaries work well. Treat a model or parser change as a reason to re-run indexing and retrieval tests, not as a transparent swap.

Keep context, metadata and provenance attached

• Headings: Store the section path and, where useful, include it in the indexed text so a retrieved paragraph retains its topic.
• Tables: Preserve each row with its column labels, units and relevant notes. Splitting a row away from its header can reverse or obscure its meaning.
• Lists and procedures: Keep prerequisites, warnings and the associated steps together when they form one safe instruction.
• Source identity: Retain document ID, version, page or section, source location, effective date and permissions needed to display a citation or enforce access.
• Parent links: Give each chunk a stable link to its parent document and section so retrieval can recover surrounding context without losing attribution.

Scanned PDFs, diagrams and complex layouts need extraction or OCR before text chunking can preserve their meaning. Validate parsed output and retain page or region references; changing chunk boundaries cannot repair missing text or a table read in the wrong order.

A practical evaluation loop

1. Build a small test set. Use real questions, the expected evidence, acceptable answer points and known tricky cases. Include tables, headings, long procedures, updated documents and questions requiring context across a boundary.
2. Create a simple baseline. Use structure-aware fixed-size splitting with a documented token budget. Keep the parser, embedding model, retrieval settings and corpus constant.
3. Change one variable at a time. Compare candidate size, overlap, semantic boundaries or parent expansion separately so a gain has an identifiable cause.
4. Measure retrieval. Track whether the required evidence appears in the top k results, its rank, duplicate-result rate and retrieval latency. Inspect misses by document type and question type.
5. Measure the generated answer. Check whether required evidence supports the response, whether claims are attributed to the right source, and whether missing or conflicting evidence is handled honestly.
6. Include operations. Count chunks and embeddings, index build and update time, storage, query latency and reprocessing cost. Verify that edits, deletions and permission changes propagate to every derived chunk.
7. Keep the simplest passing design. Adopt a more complex method only when its quality improvement is repeatable and worth its operational cost.

For a useful comparison, preserve the same data split and query set for every candidate. Do not tune a chunker against the same questions used to report its final result. Keep a holdout set for a final check, and review individual failures instead of relying on one aggregate score.

Common symptoms and what to check

• Relevant sentence retrieved without its meaning: inspect headings, definitions, table labels and neighbouring steps; test modest parent-context expansion.
• Search results contain repeated passages: review overlap and deduplication, and make sure adjacent chunks do not dominate the top k.
• Good text is missing from results: check extraction, section boundaries, metadata filters, index freshness and query wording before switching to a more complex chunker.
• Answers mix facts from unrelated sections: preserve document and parent-section identity; evaluate whether a larger context unit or reranking helps.
• Indexing cost rises without better answers: compare semantic or LLM-assisted processing with the simpler baseline and remove work that does not improve measured outcomes.

Research on document segmentation reports gains for particular approaches and benchmarks, reinforcing the need to evaluate by source type rather than generalize one result to every knowledge base. See Wang et al., “Document Segmentation Matters for Retrieval-Augmented Generation”.

Chunking is one part of the retrieval system, not a guarantee of precise or grounded answers. Keep the source structure, permissions and citation path intact; measure the evidence your users actually ask for; and prefer a simple design until a more elaborate one proves its worth.

Further reading

• Is Semantic Chunking Worth the Computational Cost? (NAACL 2025)
• Document Segmentation Matters for Retrieval-Augmented Generation (ACL 2025)
• pgvector indexing and tenant-safe filtering
• Hybrid RAG retrieval strategies