When building Retrieval-Augmented Generation (RAG) pipelines, the standard ingestion step usually looks like this:
Read a PDF file.
Dump all text content into a single flat string.
Split the string into 500-token chunks with some overlap.
Generate vector embeddings and store them in a vector database.
While this works for simple blog posts or single-column articles, it breaks down completely when applied to production documents like financial reports, scientific papers, and legal contracts.
The Three Hidden Failures of Flat-Text Ingestion
1. Table Flattening
PDF files do not store tables as HTML table elements or markdown grids. They store independent text strings positioned at absolute coordinates, combined with vector stroke lines.
When a standard parser strips away positioning, rows and columns collapse into a single stream. A table comparing quarterly revenues across three divisions becomes a scrambled sequence of numbers.…