Menu

I Profiled My Production Crawler. 95% of Its Output Was Being Thrown Away.
📰

I Profiled My Production Crawler. 95% of Its Output Was Being Thrown Away.

DEV Community: rust·ゴエンチョウ·1 day ago
#iIiFFZ4v
#devtorust#devto#dev#production#pages
Reading 0:00
15s threshold

I run AiSeek , an information platform where every published claim has to trace back to fetched evidence — a source page, a filing, a dataset. That makes the fetch layer the supply chain of the whole product: if it is slow, wasteful, or silently wrong, everything downstream inherits it.

Until October 21, that layer was Crawl4AI — a capable, popular Python framework — driving Chromium for the pages that needed rendering. I had a feeling it was heavy. Instead of trusting the feeling, I audited it: three days of production jobs (8,563 of them, October 18–20), the production database, and local reproductions run under the same limits as the production box (6 cores, 10 GB).…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.