TL;DR: I built PageSift , a single-binary web scraping engine in Rust. It's Firecrawl v1/v2 API-compatible, turns pages into clean Markdown for LLM pipelines, and just hit v0.50.0. On a live crawl of the Tranco Top 100,000 domains, its peak memory stayed flat at 188.4 MB while Spider (also Rust) ballooned to ~9.0 GB and got OOM-killed. Benchmarks, raw data, and reproduction scripts are all in the repo.
Why I built it
I run a small AI product, and web scraping was the part that kept hurting:
Self-hosted Firecrawl spins up 7 services by default and is AGPL.
Crawl4AI is Python + Playwright + Node + Chromium β an 847 MB venv before you even start.
Worst of all, captcha and interstitial pages kept landing in my database as if they were content .
So PageSift is one binary. No Redis, no database, no message queue.β¦
