If you have tried to feed a Substack newsletter into an LLM or a RAG pipeline, you have probably hit the same three problems I did:
The post HTML is full of subscribe buttons, share widgets and nested image wrappers that pollute your chunks.
Substack's archive endpoint no longer accepts offset . It returns HTTP 400, and you have to follow a cursor instead.
The archive list does not include the post body, so you need a second request per post.
I got tired of rewriting this, so I built a small tool for it and published it on Apify. Disclosure: I am the author.
Substack Scraper - Markdown Posts, Search & Comments
It uses plain HTTP requests against Substack's public JSON API. No headless browser and no residential proxies, so runs are fast and cheap.
1.…