Key takeaways

Traditional document chunking strips vital context — headings, referents, and citations — turning valid sentences into unusable or misleading facts.

Page-based PDF parsing creates high fragment rates (up to 43%), whereas sentence-based division with heading-aware stack guards reduces errors to 1%.

Effective retrieval-augmented generation (RAG) ingestion requires parse-time context retention and explicit metadata tagging before vectors are indexed or passed to large language models (LLMs).

Retrieval systems depend on a step that receives comparatively little attention. Before a document can be searched, quoted, or used to answer a question, it must be divided into pieces small enough to index and small enough to pass to a model.

\r\n The division itself is trivial to implement. Preserving meaning across it is not.

\r\n A sentence removed from page 40 of a document no longer knows that it came from page 40.…