Menu

How to export Substack posts to clean Markdown for RAG (no browser needed)
📰

How to export Substack posts to clean Markdown for RAG (no browser needed)

DEV Community: tutorial·Alejandro Robledo·about 10 hours ago
#lgXPJfIX
#devtotutorial#devto#dev#markdown#substack
Reading 0:00
15s threshold

If you have tried to feed a Substack newsletter into an LLM or a RAG pipeline, you have probably hit the same three problems I did:

The post HTML is full of subscribe buttons, share widgets and nested image wrappers that pollute your chunks.

Substack's archive endpoint no longer accepts offset . It returns HTTP 400, and you have to follow a cursor instead.

The archive list does not include the post body, so you need a second request per post.

I got tired of rewriting this, so I built a small tool for it and published it on Apify. Disclosure: I am the author.

Substack Scraper - Markdown Posts, Search & Comments

It uses plain HTTP requests against Substack's public JSON API. No headless browser and no residential proxies, so runs are fast and cheap.

1.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.