Menu

Turn PDFs into Markdown chunks for RAG in 10 lines of Python (with OCR for scanned pages)
📰

Turn PDFs into Markdown chunks for RAG in 10 lines of Python (with OCR for scanned pages)

DEV Community: python·Caspian Data·16 days ago
#lAxQxnE3
#devtopython#devto#dev#pages#chunks
Reading 0:00
15s threshold

If you have ever fed PDFs into an LLM pipeline, you know the problem: pdftotext gives you one long wall of text. Headings are gone, page numbers and running headers are glued into sentences, two-column papers come out interleaved, and scanned pages come out empty.

For retrieval-augmented generation (RAG) that hurts twice: chunks cut through the middle of sections, and the embedding of a chunk has no idea which section it came from.

This post shows a small pipeline that fixes all of that:

PDF (or DOCX / HTML) → clean Markdown with real headings and lists

OCR only for pages that have no text layer

Chunks split on heading and paragraph boundaries, each tagged with its nearest heading

A chunks.jsonl file you can load into any vector database

The 10-line version

We use the Document to Markdown Actor on Apify, which does the parsing, OCR and chunking in the cloud.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.