Menu

1 / 2

Turn a docs site into RAG-ready Markdown chunks in 5 minutes (no scraper code)

DEV Community: python·Rabii Slimane·3 days ago
#cMCVxZI1
#devtopython#devto#dev#comments#fastapi
Reading 0:00
15s threshold

You want a chatbot that answers questions about a library's documentation. The hard part is rarely the LLM. It is getting the docs out of HTML and into clean, well-cut chunks: no navigation menus, no cookie banners, no footers repeated on every page, and chunks that keep track of which section they came from.

This walkthrough takes the FastAPI documentation, converts it to Markdown, cuts it into ~400-token chunks with their heading path, and loads them into a local vector store. Total code: about 25 lines of Python.

What you need

Python 3.9+

A free Apify account (the free plan's monthly credit covers this tutorial many times over)

pip install apify-client chromadb

We use the Web Page to Markdown Actor for the conversion. Disclosure: I built it. It costs $1 per 1,000 pages converted, and failed pages are free.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.