📰00Read later0Read MoreI Profiled My Production Crawler. 95% of Its Output Was Being Thrown Away.DEV Community: rust·ゴエンチョウ·1 day ago#iIiFFZ4v#devtorust#devto#dev#production#pages+4 more🧰Tag tools✨Add tagBefore replacing Crawl4AI in my own product, I audited what it actually did in production. The framework wasn't the problem — my judgement rule, my architecture, and my exit IP were.… Read more15s
📰00Read later0Read MoreI crawled the Tranco Top 100k with a Rust engine that never passed 188 MB of RAMDEV Community: rust·ゴエンチョウ·3 days ago#fe312kFN#devtorust#devto#dev#pagesift#firecrawl+4 more🧰Tag tools✨Add tagTL;DR: I built PageSift, a single-binary web scraping engine in Rust. It's Firecrawl v1/v2...15s
